Seems like Pair Programming is “all the rage” lately in my circles. I haven’t exactly done it before but after hearing about the success and rapid knowledge growth amongst those that pair program…I was almost dying to try it! Especially after i saw David Chelimsky and Corey Haines at WindyCityRails in Sept 2009. I saw them pair and do BDD with Rspec/Cucumber and it was so fascinating, It was like I was watching a ballet as they hopped from RSpec to Cucumber and back and forth. I was like, wow…I wish I was that good! I would have paid good money for a recording of that so I could watch it again and again! I see Corey Haines traveling around pairing with people too. Some people get together and play cards, but Corey gets together to code!

So ok, I like code, I like people, I want to try it! I live a little south of Chicago so its a long commute and it seemed everyone was so busy to pair in person when I asked. I asked on Devchix mailing list for suggestions on how to do pairing online. I had found a few, and the group had some good suggestions. I even had a volunteer to try it with me! This week aimee and I set a few hours aside to try it and see if we could do it!

This article was also sort of “paired” as it was written from my perspective with input and suggestions from aimee!

We asked on Devchix mailing list for suggestions on how to do pairing online. I had found a few, and the group had some good suggestions. I even had a volunteer to try it with me! This week aimee and I set a few hours aside to try it and see if we could do it!

After introductions on Skype we set about getting a shared environment in which to code together. Ideally, we wanted some kind of desktop sharing so we could run tests, console and editor.

We had heard of a few tools and got suggestions from the devchix list:

IChat desktop sharing – we couldn’t get this to work, we did different things and it would appear to connect but then it failed. I tried to mess with settings for Sharing on mac, but nothing doing.

Rio seems to be a library to make collaborative apps, not to use in a pair programming environment.

BeSpin was hard to use.. we couldn’t figure out exactly how to use it. It almost seemed to offer to import the git repository we were working on, but then it said it only supports Subversion and Mercurial, not git.

SubEthaEdit worked but we would have to open each file individually and share each file… unless I was missing something. This would be fine for collaborating on a single file but then we could not share the test runs, terminal commands or view the browser together.

Etherpad – we didn’t end up trying this but I have used it before to debug some code or try out ideas with a friend. They recently got bought by Google, so it would be interesting to see what they do with it. This would suffer the same limitations as SubEthaEdit in that it’s just a text editor.

GoToMeeting (which is $40-50/month) its a little steep for the open source work I want to do. But people say it works really well.

VNC and Unix Screen – aimee had used this successfully before but since we weren’t on the same network, just our laptops at home, we weren’t sure it how we could make it work easily.

Then we came to TeamViewer which worked brilliantly! We shared desktop and I could type in aimee’s console window, see the tests running and type in textmate. Even with aimee on her Dvorak keyboard and I on Qwerty! I could type fine but couldn’t copy/paste with keyboard shortcuts so I used the mouse to copy/paste and it worked fine.

All in all, it was an awesome experience and I picked up on a few tidbits of knowledge from aimee on git, and rake! I had some bits of code from another project i was able to quickly copy/paste and get us rolling. We had a few discussions about coding style as we went.

Since aimee was more familiar with the codebase, she mainly wrote the behavioral specs and I wrote the code to satisfy them. We plan to switch around next time, when we pair on a different project that I’ve been developing for a while.

“Refactoring in Ruby” written by William C. Wake and Kevin Rutherford.
Published by Addison-Wesley

This is more like a “workbook” then a “how to write awesome code” book. You can download the code from github http://github.com/kevinrutherford/rrwb-code and you will find tests/specs for the exercises.

The book is arranged in three parts, The Art of Refactoring, Code Smells, and Programs to Refactor.

There are explanations of “code smells” which are one characteristic of code that could be improved. Some of them are long parameter lists, unnecessarily complex, global variable, feature envy sections, etc. One thing I find interesting is the “How did it get this way?” section. It gives some insight into the thought process and reasoning behind the smell. I think this is good, as programmers our ego may be rather miffed to hear “This code stinks” but with some reasoning, it makes the pain less and I think firms up in our minds when this happens again, to do it this other way. I always want to know why when someone says I could do such and such thing better.

In addition to the code smell examples there are three programs to refactor in the end of the book. In a conversational tone, it walks through and gives some hints on what needs refactoring. Its almost as if you had a pair programming buddy working with you and identifying in small chunks what can be improved. This is definitely something I want to work through more carefully.

What I find odd, is that not all the code smells have code examples. The inspiration for the book I think is the Martin Fowler book “Refactoring Improving the design of Existing Code” which has examples for every code smell. Maybe Ruby smells less than Java? Or those fixes are really trivial? I don’t know. Overall, this is a great book and is certainly worth the price and investment and you will be a better programmer because of it!

Our life basically lives on the web, the web is the platform! One of the advantages of creating your web application in GWT is it is pure html/javascript/css and the user doesn’t need to install a plugin to use your app! It is easily viewed on most smart phones as well. It compiles to each browser making it more efficient than using alot of if/else statments to build code for the particular browser you are using. It standardizes the method you use to make your app, no more studying the nuances and ripping your hair out over tiny browsers quirks. GWT 2.0 which is scheduled for release this fall has some great new features that I am finding to make GWT more flexible and easy to maintain.

Get GWT Trunk
The 2.0 features talked about here are in trunk and building it yourself is fairly straight forward.

1. If you don’t already have it, get the latest version of eclipse – Galileo, with the Google plugin.

   http://code.google.com/eclipse/

2. Checkout the source code
   Follow the instructions on the GWT homepage, its pretty good.
   Make sure you have ant installed. It takes awhile, i think maybe 20 mins or so on my macbook.

3. Setup the API directory in the Preferences for GWT
   In eclipse, add the new version of GWT library.

   Go to Eclipse > Preferences.
   Find the Google Web Toolkit on the left. Click add, add the path to the compile files like so:
   ~/gwt/trunk/build/lib
   I named it “trunk”

4. Create project
   Create a new Web Application Project, select “trunk” (or whatever you named it in the previous step) as the version of GWT to use.

The most interesting things in 2.0 is the UiBinder and UiHandler.

Using UiBinder
Usually for each element on the page you need to do something like this


CheckBox box = new CheckBox("this is the label for the checkbox");
box.setFormValue = "3";

Panel verticalPanel = new VerticalPanel();
panel.add(box);


If you have alot of elements on the page, that comes out to alot of java code. And you might start wondering if you are writing a Swing app or a web app?!?

in UiBinder, we can can put it in a XML file like:


<ui:UiBinder xmlns:ui='urn:ui:com.google.gwt.uibinder'
xmlns:g='urn:import:com.google.gwt.user.client.ui'>
<g:VerticalPanel>
<g:CheckBox ui:field="rubyCheck" formValue="ruby">Ruby</g:CheckBox>
</g:VerticalPanel>
</ui:UiBinder>


Typically any attribute you can use in the java code like setName, setFormValue, setStyleName you can add right in the xml as attribute=”"

You can reference your objects from the xml like so:

public class LanguageList extends Composite {
interface MyUiBinder extends UiBinder<Widget, LanguageList> {}
private static MyUiBinder uiBinder = GWT.create(MyUiBinder.class);
@UiField CheckBox rubyCheck;
public LanguageList() {
// bind XML file of same name to this class
initWidget(uiBinder.createAndBindUi(this));
// use rubyCheck as if you had defined it already with new
rubyCheck.setStyleName("rubyCheckStyle");
}
}

Note you have to bind the xml to the java code with:

interface MyUiBinder extends UiBinder<Widget, LanguageList> {}
private static MyUiBinder uiBinder = GWT.create(MyUiBinder.class);

This process is called deferred binding. This may be familar to java programmers already but My background is LAM(P|P|R) programming so I am still trying to fully understand it myself! GWT homepage explains it here: http://code.google.com/webtoolkit/doc/1.6/FAQ_Client.html#Deferred_Binding

Using UiHandler

Typically, event handlers are written as such:


// add button and handler to alert the values of checkboxes
button.addClickHandler(new ClickHandler() {
public void onClick(ClickEvent event) {
String output = "";
for(CheckBox box : boxes) {
if (box.getValue()) {
output += box.getFormValue() + ", ";
}
}
Window.alert("You checked: "+ output);
}
});

Instead of having the click handler specified in the java method, you can attach it via an annotation:


@UiHandler("languageButton")
void doClick(ClickEvent event) {
String output = "";
for(CheckBox box : boxes) {
if (box.getValue()) {
output += box.getFormValue() + ", ";
}
}
Window.alert("You checked: "+ output);
}


Sweet, huh? I think it cleans up the code nicely and gets the job done!

Go compile the Trunk version of GWT and give it a try!

If you want to see the full source for a simple gwt app before and after UI Binder, I have a write up here with links to sources on github here.

Resources for learning gwt that I found helpful:

GWT Homepage
http://code.google.com/webtoolkit/

GWT Can Help You Create Amazing Web Apps
http://www.youtube.com/watch?v=T0X9BcBd-I0

GWT Best Practices
http://code.google.com/events/io/2009/sessions/GoogleWebToolkitBestPractices.html

OnGWT – various blog posts about GWT
http://www.ongwt.com/

Published by Apress
By Kevin Marshall, Chad Pytel, Jon Yurek
Book Info
Sample Chapter: Ch. 01 – Introducing Active Record
Table of Contents

Years ago when I was in PHP Land (now I travel quite a bit more! haha), I strugged for months with how to write a good ORM . It was tough, because I wanted to abstract the “boring logic” of retrieving records from a database without writing SQL but still remain flexible enough. I never really came up with a good model. I used the DAO from “extreme php” library which I think was a knock off from java. It was ok, but I still didn’t feel like I had “arrived”.

When I discovered Ruby on Rails, I found ActiveRecord. Ahh HA! Finally, this is what I was looking for. At first I thought it was part of Rails, but its not. Its a standalone library and you can use it with straight up ruby scripts.

I got a review copy of “Pro Active Record” some time ago and read it some when I got it, then some later, and now I am going to officially write up a review!

If you do anything with Active Record, get this book. The things that are briefly mentioned in most Rails books are described in detail in this book.

Chapter 1 – Introducing Active Record

Most of the time, the first chapters of a book are boring to me. I don’t need another “History of the Internet” or how “HTML was developed” … blah blah. But this one, the story is only 1 page. And it actually has some introductory scripts on using Active Record, so you can see right away how it works. It also explains the benefits of MVC and why ORMs are good. Some people still don’t get it!

Chapter 2 – Active Record and SQL

This chapter helps you translate the “sql in your head” to how to write it with Active Record. I’ve used Active Record so much that now I have forgotten most of my SQL, which is kind of embarrassing. I now find writing sql tedious and boring! I would have actually called this chapter “Demystifying Active Record” since it explains why all the dynamic finders work. You’ll also find transactions and locking explained here.

Chapter 3 – Setting up Your Database

Migrations! The Awesome Thing that can turn into a nightmare for large rails projects with multiple developers…. definitely have to decide on some best practices with your team on this one. The chapter has only one thing to say about this — assume any checked in migration has already been run by your team and the migration should not be edited and checked back in! You’ll have to make another migration file with your changes.

[tip]
Nola's Note: When you make a migration, test it both UP and DOWN!! Here's what I do --
write a migratiion
rake db:migrate (go up to the version with new code)
rake db:migrate VERSION=n-1, (go to version before the latest)
rake db:migrate (back to lastest)
rake db:migrate VERSION=0 (back to blank db)
rake db:migrate (back to latest)
[/tip]

Just to be sure its all good — even on a new database!

Chapter 4 – Core Features of Active Record

Now is the fun stuff – Callbacks. This is magic. This makes Active Record so flexible, and is one thing I could never figure out how to do with my PHP ORMs. I use call backs to set defaults for fields. If its just a straight default, then I set it in the database but if I need to make a decision, (if this field then this field..) then I can use it in a callback.

Associations – at first this is very confusing! I don’t know how many times I got “has_many” and “belongs_to” mixed around in the beginning.

Validations – Awesome. I had to do some ruby code without a database and I really really really missed the validations. It took me like 5x longer than it should! Understanding all of these validation methods will make your life so much more enjoyable. I really really hate doing boring, repetitive stuff…it seems so wasteful to me.

Chapter 5 – Bonus Features

Everybody likes a bonus and this isn’t even the last chapter of the book.

Java people will like the Active Record Observers — seems a little AOP to me (aspect orienteted programming) and something I probably have neglected to use to their fullest extent.

Acting up — Learn how to “save time” with the “acts_as” magic: List, Tree, Nested Sets. If your data needs these structures, you got it made. I can imaging how much longer it would take to write this stuff in perl or php.

Composed of – I haven’t used this, but this looks like a good way to make sensible objects out of database tables. There is quite a bit of explanation and examples of this, it will come in handy.

There are a few other in depth explanations of things, such as method_missing which is how alot of the magic happens. Rock on.

Chapter 6 – Active Record Testing and Debugging

Ahh yes, Testing. My favorite subject. My friends who know how much I love testing say I am sick. I must have an inner need to PROVE I am right or something, haha.

The chapter goes into depth about using test_unit with Active Record, sadly no RSpec. But, it does go into all the error messages that Active Record throws so you can write good try/catch blocks and make very exact error messages (probably best logged for the admin rather then displayed to the user!)

Chapter 7 – Working with Legacy Schema

Here’s how you work with that old database that just won’t die… or that management won’t let you totally redo. Active Record follows some of the principles of Rails “convention over configuration” … relying on table and column naming conventions to figure out how to build your object….but still giving you a way out if you want your tables singular and your primary id field called “myawesomeid” instead of “id”

I’ve used some of these things on an older database and it was possible! Not too bad if thats what you have to work with.

[soapbox]
Some people find this annoying "oh gosh! my library can't make decisions for me! OMG! That sucks" .. to that I say, "Umm ok. But if you follow these conventions then I can come into your project and know exactly what is going on" ... like with web standards, we all harp on how IE and FF do things differently, yet people want to bellyache about Active Record preferring to have plural names and id field called "id". Right.

Follow the dang convention and find something worth complaining about to complain about.
[/soapbox]

Chapter 8 – Active Record and The Real World

This chapter goes into depth about the library and encourages you to go read the Active Record code. Always a good idea to know what it is you are using I’ve actually learned ruby better by reading source code. The chapter walks you through basic structure of the files. Very cool.

[soapbox]
I used to work at a place that didn't like any "outside code" because they were afraid "OMG ... it will send our passwords to Russia!" ... ok, well I am not an idiot. I read over any code that I use that I didn't write. I look at the tests to see if I am using it right. I even RUN the tests so I can be sure its working as advertised.
[/soapbox]

Alternatives to Active Record – with EXAMPLES! If something about Active Record doesn’t set too well with you, take a look at the alternatives. Sometimes I look at the alternatives and decide that the first wasn’t so bad after all. You’ll find examples of DBI, Og, ActiveRelation.

Finally a section on Q and A finishes up this book. The Appendix has a complete reference of ActiveRecord methods to make this book a well rounded reference, tips, documentation and very handy to have at your desk!

Book Site | Sample Chapter: The Theme System | Table of Contents

Many of you are aware of my current total infatuation with Ruby, and that I’ve used PHP for about 6 years and at one point decided I hated PHP…until, I needed it for a quick one-off page and then realized that PHP had its place. Then again, I was totally frustrated with Ruby when making my moms bakery site and then turned to Drupal and Gallery (another fine PHP project), which saved my bacon and I got a website and photo gallery up in a weekend. So, PHP and I have had our moments but I’m not abandoning it!

Drupal powers some big sites, its not just for joe smoe’s blog. This is an interesting page about Is Drupal Right For You? and if you are wondering if its something that would even work for you.

I was excited to get my hands on a review copy of Pro Drupal Development. Its no secret that coders hate documentation and Drupal has one of the most complete online documentation I’ve seen for an Open Source project, but its almost too hard to find what you need amongst so much. The Pro Drupal Book is a godsend for the drupal programmer, new and experienced alike. I wish it was written a year ago!

The book starts off with a quick overview of how Drupal is structured and defines terms such as hooks, node and blocks in just 10 pages. Chapter 2 is a A step-by-step tutorial with making a module. That is a great idea to start off quickly writing code. It get the reader involved and hands on. I really tire of books that have to start off with the history of the internet, html and how things have evolved. Get to the code dangit!! Kudos to the Authors for that! Chapter 3 gets into module specific settings, like how to get your module to show up on the admin page and storing user settings that your module needs.

After you’ve had some experience with the code then the book goes into details on the specific parts of Drupal:

• Menu System
• Databases
• Users
• Nodes
• Themes
• Blocks
• Form API
• Filter System
• Searching and Indexing
• Files
• Taxonomy
• Caching
• Sessions
• JQuery
• Localization
• Using XML-RPC

Drupal is a pretty amazing framework, when I read the code I say “why didn’t I think of that?” … the module and hook system is genius.

Then some more general topics:

• Writing Secure code
• Development Best Practices
• Optimizing Drupal
• Installation Profiles

One of the chapters I skipped ahead to read was The Form API. In my years of PHP I’ve often tried to come up with a framework for doing forms and I wanted to see how they did it. This chapter follows a tutorial style as well. The Form API allows you to define fields, their label, their value, description. Some frameworks take the template approach, where you hammer out your HTML. Some are more configuration based like Drupal making a multi-dim array with keys and values. I can see advantages to both. There is a hook function for validation which allows you to write your validation checks.

PHP gets a bad wrap for security, partly because its pretty easy to learn PHP and newbies don’t always realize what they are doing. There is a chapter devoted to security and includes even some things I didn’t know about — encoding mail headers. The Form API is very secure,  one thing it does is check values that come from dropdowns were actually in the options and it wasn’t something that the hacker made up.

Developer Best Practices are great for the new developer, it talks about using cvs, tags, branches. It talks about how to create and apply patches (hint – you can contribute back to drupal). That is awesome. Alot of open source projects are like “HELP us, submit patches!” and the new user is left with uhhhhhh..how?

Caching is another interesting chapter. You will learn  how caching works and how Drupal Core uses it. There is a Cache API that has methods for module creators to make their modules faster.

JQuery … I am not sure if I like it or not, but its part of Drupal 5! I skipped ahead to this chapter to see what its all about. There is a javascript hook built into Drupal making it easy to add, thats pretty cool.

One thing I found lacking in the book is anything about Testing. There are few pages on debugging and some modules to help with testing, but I would like to see more. At least some talk about selenium, which is great for a site made with any framework/cms.

Over all, Thanks APress for another great book!

HTML tree, that is… Ahh HTML. The tree with often not so perfect branches. Recently for one of my projects I had to grab certain bits of information from a series of HTML pages. The HTML was proper in some and not so good with others (huh? Font tags? Poorly nested tags?). I thought about parsing it as I would an XML document but given the “crooked branches” I figured that wouldn’t work. I could write a mess of regex, but gee — isn’t there a better way? Through a series of searches and poking around, I discovered HTML::Tree module which seemed to suit my purpose well! I will not bore you with the use I had for using HTML::Tree, so I made a fun sample:

#!/usr/bin/perl

use warnings;
use strict;
use diagnostics;

use HTML::TreeBuilder;

The typical top of a Perl script. I usually ″use diagnostics″ when developing and take it out when I am done. Just give you more verbose error messages. I need all the help I can get!

Now the fun stuff:

I made a function to load the DevChix homepage that I had saved to a local file. Ideally, this data would be pulled live from the site. For now though, I start simple.

sub load_tree {
   my $page = HTML::TreeBuilder->new();
   $page->parse_file('DevChix.htm');
   return $page;
}

This returns the TreeBuilder object with my data loaded. Using the most awesome tool Firebug (Also, see Jen′s post about it awhile back), I see that the sidebar list is a div tag with id of “sidebarposts”. Lets look down our Html Tree and find that element:

my $page = load_tree();

my $sidebar = $page->look_down( '_tag', 'div',
   sub { $_[0]->id eq 'sidebarposts' } );

Not too complex. Look down the tree, look for a tag thats a div with the id of “sidebarposts” .. gee, thats nearly english (and people say that Perl is jibberish! Bah!).

Now, lets grab the li elements in that div:

my @ul_list = $sidebar->look_down('_tag','li');

foreach my $li(@ul_list) {
  print $li->as_text, "\n";
}

I know I′ll be getting back more than 1 element so I assign it to an array instead of a scalar. Then in the for loop, I want to iterate through the list and print the element as text, which gives me the name of the link.

Output is something like this:

...
Regular Expressions: The Wart on the Bum of Every Language in Existence
RUBY: DRY up your Enumerations
*waving, not drowning*
Beautiful Python: The programming language that taught me how to love again
RailsConf
Test More for Java?!
Book Review: Beginning Ruby On Rails E-Commerce
...

Using a code ref to find elements came in extremely handy when I had some bunked up HTML, sometimes I had to look_down a table, find a tr that contained a table which had a certain class, or find all href tags that had a certain domain in the URL and then grab the tr container so I could grab the following tr .. etc.

The Tree::Builder interface has a ton more methods than I’ve talked about and you should take a look at it next time you need to grab bits from html pages!

At the 2007 Rails Conference, Geoffrey Grosenbach caught up with a bunch of the DevChix and did a round table discussion about Women in Open Source. We have alot to say, so this is the first of two episodes!

Geoffrey — thanks for the consideration in ordering women’s tees. You Rock!

Thanks PeepCode Screencasts for sponsoring the podcasts.

Any one know, does there exist a TestSimple/More (from Perl) for Java like the module for Perl?

whats that you say? I wrote about TestMore in Perl and PHP on my blog awhile back: here and here also!

I did a quick scan with google didn’t give me much hope. So I wrote this quickly on the train this morning, since I can’t hardly stand to program in any language without a TestMore like implementation.. (I know there’s JUnit, and I’ved used it before, but its alot of overhead.. I just needed some quick tests!)

Class: (not making any claim this is the best or greatest way, and not 100% TAP protocol)

package com.myawesomesite.util;

public class TestSimple {

 private int testCount;

 public TestSimple() {
  this.testCount = 0;
 }

 public void ok(boolean truth, String message) {
  this.testCount++;
  System.out.println( (truth ? "ok " : "not ok ")
                      + this.testCount + " - "
                      + message);
 }

Usage:

package com.myawesomesite.java;

import com.myawesomesite.java.*;
import com.myawesomesite.util.TestSimple;

public class TestAnimal {

 public static void main(String[] args) {
  TestSimple test = new TestSimple();

  Dog bob = new Dog();

  bob.setColor("green");
  bob.setCollarSize(10);

  test.ok(bob.getColor() == "green", "bob's color is green");
  test.ok(bob.getCollarSize() == 10,"bob's collar size is 10");
  test.ok(bob.getCollarSize() == 0,"bob's collar size 0");

 }

}

output is:

ok 1 - bob's color is green
ok 2 - bob's collar size is 10
not ok 3 - bob's collar size is 0

Beginning Ruby On Rails E-Commerce
From Novice to Professional
by Christian Hellsten and Jarkko Laine
published by: Apress

Book Site | Sample Chapter | Table of Contents

I got this book to review and set it on the shelf for a few months… by the time I got to it Rails was up til version 1.2 and this seems to be written for version 1.1.2 – DOH! I tried a few examples and wasn’t compiling. After a little investigation there are only a few differences that would hinder this book from working with Rails 1.2. Namely, the assert_select has replaced assert_tag. That being said, this book is still great and applicable to Rails today. If you think about it, with as fast as Rails as grown it is impossible to keep 100% up to date!

This book is totally fantastic for beginners – because it actually shows Test Driven Development. What you say? Most books say something like that “to keep code size down, tests and error checking have been left for an exercise to the reader” … Riiiiiiiiiiight. How are you going to teach people coding that way? Tests should just be an automatic task of a programmer. Write some test… write some code. I honestly can’t imagine anymore how you could code anyways without them!

Not only does this book cover testing (including acceptance testing with selenium in later chapters – whoo hoo!) it starts out with not using scaffolding. I think, and this happened to me, at first I used scaffolding for everything and didn’t really understand the process. The book first goes through the “scaffold process” by hand, writing each method and view – after writing the test. Very cool. Then it tells you how to use scaffolding for the next model in the sample application. Awesome.

It talks about common concepts for really any site – tagging, adding forum, adding a form to upload images, browsing a list of products, multiple language support. Even if you are not selling anything on your site, you will still find this book extremely helpful.

Impress your friends! Learn how to write a DSL for testing. This is cool stuff, DSLs fascinate me to no end. Any and all mentions of it I study intently. Rails is in a way, a DSL for web applications!

Being true to the title “e-commerce” it actually talks about how to do payments over the web. Most books who talk about shopping carts skip that important step!

When you are ready to make your millions on the web there is quite an extensive chapter on deploying your site. It talks about LightTPD, capastrano, caching, and security! Its really nice to have all this in a book, instead of constantly looking online for documentation

My only complaint is – it doesn’t specifically mention what version of rails it used, I assume from the output of script/about that is 1.1.2 … and they should of talked about how to check out a particular version of Rails, just in case you wanted to use the exact version that is used in the book. Which may not be a bad idea for new users.

On one of my projects the requirements was to take two csv files, one with words and data and one with words. The goal is to use the list of words to retrieve the data from the other. Long-term storage for the datasets is not needed, so I didn’t see a reason to put in a database. Typically the language used for this company is PHP (though I have sneaked in a few ruby scripts.. shhhh) so my first try was to load each file into an array in memory and do a lookup. It worked fine for my sample files but croaked when I fed it the real data. My real data file was 180k some words and my word list was 400k words. So, I tried using a temporary table. I started it when I left for the day and it was still going the next morning. This was unbearable slow and never actually completed. I could have probably tried to optimize or choose different table types, but I didn’t…because I was thinking hmmmmm… wonder if Perl will be faster? It was, not only did it complete — but finished in minutes (with printing each word out to the display) or in seconds (in a more silent mode). I showed the people who would be using it – they were amazed. This sort of thing takes hours in excel and crashes sometimes.

# load data list into hash
$parser = Parse::CSV->new( binary => 1,
                    file => $infile_name );
my %DATA = {};
print "\\n\\nNow indexing data file\\n";
while ( $line = $parser->fetch ) {
	next if !$line;
	$word = shift @$line;
	next if ( (!defined($word)) or ($word eq ''));
	$DATA{$word} = $line;
	print $verbose ? "Indexing $word\\n" : '.';
}

The files sometimes have foreign characters in it so I open it in binary mode. Parse::CSV is a wrapper for Text::CSV_XS class and is much easier to use. Its written by Adam Kennedy and I had a question about it and was able to find him in IRC and ask him myself, and got some help from someone else in the room too — pretty cool! Back to the code, I skip if the line is not defined, shift off the first element of the array. The fetch method always returns an array reference, so I must prefix it with @ to access it as an array and shift off the first element. If thats not defined or blank, I use as the key to store the reference to the array. If verbose option was used, I display a longer notification to the user, otherwise I display just a period. That way the user knows its working!

#setup excel file
my $workbook = Spreadsheet::WriteExcel->new($outfile_name);
my $worksheet = $workbook->add_worksheet();

#process word list

$parser = Parse::CSV->new( binary => 1,
                 file => $wordfile_name );
my $row_count = 0;
my (@row, $row_ref);
print "\\nNow starting lookup...\\n";
while ( $line = $parser->fetch ) {
	@row = ();
	$word = shift @$line;
	next if ( (!defined($word)) or ($word eq ''));

	if (exists $DATA{$word}) {
		#word found
		$row_ref = $DATA{$word};  #get ref to array
	  	print $verbose ? "Found $word\n" : '+';
		push @row, $word;
		push @row, @$row_ref;
		$worksheet->write_row($row_count,
                                      0,
                                      \\@row);
	} else {
	  	print $verbose ? "Not Found $word\n" : '-';
		$worksheet->write_row($row_count,
                                      0,
                                      [$word, 'not found']);
	}
	$row_count++;
}

Similar to the loading of the hash, I open the file with Parser::CSV as before. I initialize my variables and set a row_count used to write to the excel file. This data file is supposed to be a single column list of words, so I could probably get away with just doing a simple file read but just in case I treat as a CSV file. Once I get the word variable, I see if it exists as a key in my %DATA hash. Funny thing about Perl regarding hashes and arrays – when referring to the whole thing you use % for hash, @ for array. But when referring to a particular element, you use $. Kinda funny but in a way makes sense…hehe. I simplified this code a bit so you get the concept and these lines better no doubt. I used @row as a temporary holder for the data I want to write to the excel file, I had some other code here I took out which had more to do with the @row. Finally I write the row array to the the excel file. The method requires an array ref, and \@row gives me a reference.

Thats basically it. The loading of %DATA hash is of course, dependent on how much memory you have. If you have problems you could try tieing it to a DB file:

use DB_File;

tie %DATA, 'DB_File', 'output.db'
         or die ('cannot open output.db');

Building a DB file also has the advantage of persistence. If you need to run a lot of lookups, multiple times you may be interested in using it. It will be slower to some extent, but it will work!

My script could probably be optimized some and perhaps golfed some which could make it faster. But I try and write Perl in english, at least for the first pass. Then perhaps I can use the shortcuts and see if it runs faster. This was a fun project and I got to learn more Perl. My friends Liz and Yaakov helped me and I thank them