JavaScript InfoVis Toolkit

The JavaScript InfoVis Toolkit is another example of a library exploiting Canvas technology. I put together a small demo Rails application that uses the toolkit to provide a dynamic tree view of the Rails application directory structure.

Don't Miss Aloha on Rails!

You don't want to miss the Hawaii Ruby On Rails Conference - Waikiki, Oahu, Hawaii - October 5-6 2009. The speaker list is impressive and still growing. The venue is awesome and is only a short (really short!!!) walk from Waikiki beach. The economic downturn is making flights to Hawaii very reasonable right now. This conference is being primarily organized by Seth Ladd, a Honolulu based RoR developer and great co-worker of mine. There is an Aloha On Rail Blog featuring the latest information. The latest post features another great co-worker of mine, Honolulu Hacker Kevin English in a clip on why you should come to Aloha on Rails. Why Aloha on Rails??? mingle with a group of very talented developers gain insights on the latest technologies and techniques re-charge your batteries reward yourself! This is a no brainer - register now!

Rails Bug in *_changed?

UPDATE: The latest rails has fixed this problem but this problem was still present in Rails 2.2.2.

The new dirty methods in Rails show this interesting result:



The above code has position as a nullable integer column. When parameters come in a post via the parameters hash position comes across as a string. For 14 you get the Rails goodness and assigning "14" is not seen as a change - but why not the same thing for 0?

The problem is in the following offending code in vendor/rails/activerecord/lib/active_record/dirty.rb file. I have commented out the problem line (look for the no!!!!...) and added what I believe to be the correct code below it:

Rails ActsAsList - Initializing position Values

Recently I worked on a project that had a model acting as a tree with the children acting as a list within the scope of the child's parent. The old school ActsAsTree and ActsAsList plugins used in combination, handle this scenario nicely. There was one problem however - we had a large "tree" to load in where only some of the children were ordered under their parent. The rest of the children had a NULL value sitting in the position column. Because ActsAsList works by setting "position = position + 1 WHERE position >= x" when insert_at(x) is called this wasn't going to work with the data as is. "NULL = NULL + 1" just doesn't seem to work out too well. I decided to pre-process the data putting the children in their natural position order. In this case the natural position order would be by the provided position with a secondary sort on short_name if the position was not given. Also, when a child list contains both numeric positions and NULLs the NULL children should go to the bottom. Given the number of rows I was processing I knew this was going to take some time so I decided to see how this would perform when implemented as a MySQL stored procedure. Here is the code I came up with shown in the migration that creates the stored proc:



And here is the rake task to call it:



The stored procedure took just over a minute to do the renumbering. After implementing I decided to compare this to a straight SQL version that would be portable to different databases - but not as fast. I tried to avoid an ActiveRecord tax here and did the SQL in raw execute statements. Here is the rake task:



The final results showed the stored procedure technique to be about 5 times faster than the SQL technique.

Quicker Rails Seed Data Loading

The word is that Rails 3.0 will feature a way to load seed data. This is sure to be a handy and needed feature, however, when loading large amounts of seed data you are probably going to have to abandon ActiveRecord and / or fixture style loading in your db/seeds.rb file in order to get the kind of performance you want. Recently I set up a way to load seed data for a Rails 2.2.2 project which exploits the "LOAD DATA INFILE..." command that MySQL provides. This cut the data loading time for my particular data set from over 15 minutes (I gave up waiting) to less than 30 seconds. This technique is likely to remain relevant for your future db/seed.rb files. The following rake tasks set up my Rails application to load ".psv" and ".yml" seed data files from the db/seed directory. The ".yml" files are normal YAML fixture files and are loaded via my rake tasks using an ordinary fixture technique. The ".psv" files are pipe separated files which are loaded use the above mentioned "LOAD DATA INFILE..." command. The way I have this set up here the order of the columns in your ".psv" needs to match the column order in your database so you may want to tweak this code a bit and provide parameters to specify a different column order. In other words, YMMV, anyway, code follows:



One thing to look at carefully when using this technique is what your database product does with "empty" values in your data set. MySQL didn't want to seem to let a empty numeric value be NULL which caused me to go with some data massaging before and / or after load.

DSN-less ODBC Connections on Linux

On the Rails Wiki Connect To MicrosoftSQLServer From Rails On Linux Box page a database.yml sporting a DSN-less connection is shown like this:



That won't work on my Linux box given the way unixODBC and FreeTDS installed themselves. Looking at that you might think the quick solution is just to correct the path to the driver file but that won't fix the problem either. This is where the head scratching starts. Apparently when installing on Mac OS X ODBC puts an entry in the /etc/odbcinst.ini that is labeled [/path/to/the/driver/file.so] (or maybe that setup actually works with a path, but I don't think so). On my Ubuntu Linux box, the only entry in /etc/odbcinst.ini is labeled [FreeTDS]. So getting the DSN-less connection to work is as simple as using:

Chad Fowler's 20 Rails Development No-No's

Chad Fowler, who will be keynoting at the Aloha on Rails RailsConf, recently released a list a 20 Rails Development No-No's on his blog. It is a decent list to review to help make yourself aware of various Rails "code smells." Throwing in my 2 cents, for number 10, Code in the Wrong Place, I would like to mention controllers that load the flash with HTML. If you are putting any HTML in your flash then you should use a proper view technology, such as a partial, to load it up:



Ok, so thats a pretty trivial example, what I really wanted to say in this post is that Aloha on Rails is coming and it is going to be pretty awesome. Seth Ladd released a nice blog post about it today. You should really check it out.

Rails - Read More, Code Less, do Peer Reviews

For some previous projects I rolled some simple code for generating a random string - used to generate a password reset token for example:



Now I've found that this type of functionality has been in Rails for some time - now as ActiveSupport::SecureRandom and previously as Rails::SecretKeyGenerator.

Ok so that is simple code and probably isn't a problem but the lesson here is: read more, code less, do peer reviews...don't waste a lot of time re-inventing the wheel.

Rails ActiveRecord Order by a Function

Of course you can pass functions in the order clause through ActiveRecord finders:



In the above case a Foo has fiscal_year and fiscal_month columns unfortunately stored as VARCHAR(255) containing, for example "2008" and "7". While the sorting will work that query will be able to leverage an index on these columns only if the database used can create indices that include functions. I'm not sure which databases can do this. From a quick look at the SQL Server and MySQL documentation it doesn't appear that either of these databases can do this.

Rails Caching and MySQL GROUP_CONCAT

A recent caching solution that we came up with involved using the hash of the concatenated "children" ids of another resource to identify when the identical and previously stored children payload could be delivered instead of constructing the payload from scratch (which is kind of expensive). Our current solution uses action caching and a custom ActionController::Caching::Fragments::FileStore. In our custom FileStore we override the real_file_path(name) method to use this "children signature" as part of the name. The class looks something like this:



This seems to work well and the next step is likely to look at using the children_sig value for the ETag as well.

One interesting thing that came out of this investigation was that I wondered if I could produce the hash using straight SQL - should be much faster right? Turns out that with MySQL there is a built in aggregate function that concatenates a column and it is called GROUP_CONCAT. To generate that hash in the database with MySQL the following will work:



Pretty cool! I'm not sure which other database products support this as a built in aggregate function but it doesn't appear that SQL Server does.

I found the following recent blog posts talking about other uses for GROUP_CONCAT as well:

http://tempe.st/2009/03/the-thrill-of-a-new-technology-couchdb/
http://www.christianmontoya.com/2007/09/14/mysql-group_concat-this-query-is-insane/
http://db4free.blogspot.com/2006/01/hail-to-groupconcat.html

Rails Select Filters that Map to In Queries

Example of a Rails select box that performs filtering and is backed by an IN query. First a constant is defined in the controller. It is an array of arrays where the "first" is the text to display in the select and the "last" is an array of the values the first represents.



Then we render that in our page with typical code:



Back in the controller, when the form is submitted, the following code sets up the conditions for a find call:



The only somewhat interesting bit is how the conditions for "Other" are set up with an inject from the valid options defined in the SERVICE_GRADE_CODE_OPTIONS constant.

Apache Tip: Encourage Browsers to Open Documents in the Native Application

Jakob Nielsen says that you shouldn't open "non-web documents" within the browser window. Some of our users had the same complaint and were asking for us to prevent pdf, doc, and other file formats from opening inline within the browser window.

The way to get this to happen is to add a "Content-disposition" header with the value of "Attachment". Of course if you are using Rails send_file or send_data methods you can easily add this header. If, however, the files are being served up directly by your web server then you need to find another way to add this header. If you are using Apache 2 the following directives in your conf file should do the trick:

Send HTML to Javascript Functions from Rails Partials

Many Javascript libraries, such as ExtJS, allow you to pass HTML to functions. A good example of this is the Ext.Msg.alert call which accepts a a title and a message where the message string can have HTML markup in it to form the content of the alert popup. When this message is a large block of HTML, for example a large disclaimer that you have to stick up in the face of your customer, it becomes very awkward to include this HTML text in the same page that shows the popup. It bloats the page and mixes together what would be better edited by a "subject matter expert." What you really want to do is to put this disclaimer into its own partial but then you have to figure out how to render the string so that newlines and other content don't break the Javascript. How do you do that? Here are two ways that work:

attachment_fu with Optional Attachment...again

I wrote about this previously...attachment_fu is not mean to be used to provide a file attachment as an optional attribute of a model with other attributes. If you want to combine a model with its own attributes and an attachment have a look at Paperclip. attachment_fu is a great plugin but the attachment is supposed to live in its own model.

So why am I writing about this again? Well, recently a project that I had worked on before where someone put the attachment directly in another model was updated and then had some plugin problems. I solved the problems by upgrading some of the plugins, including attachment_fu. This upgrade resulted in further breakage of this attachment_fu within another model / optional attachment problem. Now, attachment_fu calls attr_accessible :uploaded_data from the has_attachment directive so you will have to open up the model for your other attributes for bulk assignment. This will look something like this:



And as before, the plugin must be patched to prevent errors that would occur when updating or deleting a model that doesn't have an attachment. The patch is here which works against the current attachment_fu:



Now the real question is, why haven't I either 1) fixed this the right way or 2) switched to Paperclip by now? Lack of time and an already existing file structure laid out by attachment_fu I suppose...but this is getting ridiculous.

Ext JS Custom TreeNodeUI

I wrote this some time ago and then noticed that Blogger blasted the indentation in the code snippets so I'm only posting it now.

Hey, this should be a good dump - I needed an Ext JS tree with a custom TreeNodeUI. In addition to the usual text, I wanted each TreeNode to feature a select box that would allow a selection (and Ajax saving of the selection) to be made for each node. While the Ext JS API Documentation is pretty good, it doesn't always provide the information needed when trying to "step outside the box." (This framework is just dying for someone to write a book about it...) After just a little head scratching I found writing such a custom TreeNodeUI to be fairly straight forward.

The first thing I needed to do was to define my SelectNodeUI having it call its parent constructor on instantiation:



After that, I needed to extend my SelectNodeUI, inheriting from the standard TreeNodeUI, while overriding the renderElements and onSelectChange methods. The renderElements method is overridden to add the select box (look for the sel variable). The onSelectChange method is overriden to prevent the normal focus change which would unfortunately cause the select box to close immediately when first selected. This looks like a lot of code but is primarily a copy of the TreeNodeUI renderElements method with the rendering of the select box added. I've marked the parts I've added / changed with // DJB: comments:



Take special note of the building of the selectHtml string from the COMMAND_HIERARCHY_TYPE_ARRAY which has been written into the page in the format of [["label text", id], ...] and how the correct option is selected after the selectHTML has been rendered into a select box DOM node.

Now that the custom tree node is defined and ready to use the loader for the TreePanel needs to be informed about which UI providers are available. This is done by setting the uiProviders property of the loader property of the TreePanel:



With this code in place, a mapping has been defined between the string "select" and the SelectNodeUI. If the JSON for a tree node comes back with the value uiProvider: "select", the SelectNodeUI will be used to render that tree node, otherwise, it will fall back to using a regular TreeNodeUI. Keep in mind that such a tree node will also need a typeId property so that the proper option can be selected in the select box. With Rails the JSON for these tree nodes might be rendered with a method like this:



Unfortunately, as the comment says in the code, Internet Exploder does not bubble change events. Thus each select node has its own change listener that calls selectChange (otherwise I could have wired up a single change listener at a level above these tree nodes in the DOM tree). selectChange is a fairly straight forward function which uses an Ajax call to save the new state for the tree node's underlying model:



Finally, the select boxes are slightly taller than the normal content held at a tree node and thus mess up the lines that are drawn connecting the + / - expanders along the left side of the tree. Instead of fixing this up to look write I would recommend just turning the tree lines off. This can be done with by setting up the TreePanel with either the lines: false or useArrows: true properties. With lines: false the tree will render with the same old + / - expanders but won't render the lines. With useArrows: true the tree will render with a look that is supposed to emulate "Vista," using small triangles and drawing no lines. I'm not sure what this "Vista" thing is and I couldn't seem sudo apt-get install it ;-).

Analyzing Rails Log Files for Slow Actions

There are plenty of packages for analyzing rails log files. These are helpful when you want to find bottlenecks in your application. A very easy to use one is rawk.rb discussed in Railscast #97. These tools are great as they help you to spend your efforts on the most costly areas of your code.

However, if you want to do a really quick, but less precise analysis of your code you can resort to some simple Unix tools as shown here:



This simple command line greps out the "Completed in" lines, cuts out the 3rd and 17th columns as separated by spaces, and then sorts the result. The two fields that have been cut out represent the number of seconds and URL for the action in question looking something like this:



9 seconds! Ouch!

Of course you can get much more advanced in the command line department with tools like awk but at that point you might as well download rawk.rb and get back to work.

ActiveRecord Change SQL Server Query Timeout

Recently I had a rake task that was timing out while moving some data from a fact table into another data table. This task has to be run only when we get a new batch of data, do some ETL work, and then dump the data into the production database tables. I found out how to change the SQL Server query timeout on the Backyard Bamboo blog. Since I just wanted to turn off the timeout for the current session I ran some code like this:



The 0 turns the timeout off, otherwise the value is the number of seconds to set the timeout to. The default appears to be 30 seconds.

The timeout probably wouldn't occur if we add an index to a certain field on our fact table but this will get our data into production until we can make that change in the ETL process.

SQL Server information_schema.views Length Limitation and rails_sql_views

Recently I was running into a problem with a plugin called rails_sql_views (part of the Active Warehouse project). This is a great plugin created by some friends of mine that helps you manage views in a rails project. The plugin manages the proper creation of the view in the schema.rb file. For the SQL Server adapter the code to produce view comes from a query that looks like this:



The problem was that the view queries were being cut off. The cut off point seemed to be around 4000 characters. I mentioned this to my colleague Semergence and he recommended I look into the column definition for the view_definition column in the information_schema.views table. Sure enough, the column was defined as a nvarchar(4000). Who would be crazy enough to create a view that required more than 4000 characters right? Well, errrr, me I guess.

So armed with this 4000 number my google searches became very effective and turned up this from Joseph Scott. In the blog article Joseph reveals the sp_helptext stored procedure. Using this stored procedure you can easily get the full text of a views create statement, however, you do have to reconstruct it as it returns the result in separate rows under the key "Text". Here is the simple code:

attachment_fu with Optional Attachment

Note: attachment_fu has changed a bit, see the update to this post here.

attachment_fu is the most popular plugin for providing file upload to a Rails application. One thing you need to be aware of with attachment_fu is that it is meant to be used by making a model just for the attachment that will be the child relationship in a has_many or has_one. An example would be a User model that has such a relationship with an attachment_fu Avatar model. If instead you put the attachment_fu model directly in the User class and then allow that attachment to be optional, you will run into two problems:

1. on update without an image fall down go boom
2. on destroy without an image fall down go boom

Recently we had an application where this mistake was made and the problem wasn't noticed until the application was deployed and people started creating models without an attachment - fall down go boom.

It is, however, possible to patch attachment_fu so that it will work as an optional attachment. Information about such a patch can be found here. The provided patch only takes care of the problem of updating a record that doesn't have an attachment but the comments suggest a solution for the problem of destroying such a record as well.

With that said, Paperclip looks like the new kid on the Rails file upload block designed to avoid such problems. I think for my next file upload requirement I'll definitely take a look at Paperclip.

Ruby Singleton Methods Handy for Quick and Dirty Debugging

In a previous post I talked about needing to dump ActiveRecord models in favor of record sets when producing large reports from a Rails application. While this works well, the record set returned when doing a select_all is an array of hashes. And, being a hash, if you misspell the column name and type, for example:



nil will be returned and you will be none the wiser. Rubies singleton methods can quickly be used to detect such mistakes. Merely override the [] method with a method that will raise an error if the key can't be found:



By appending the singleton method override for [] you can quickly find out if you have made a typo somewhere in your view when specifying the column names of the record set.