self.d-struct.org

Today is my last day with IBM Interactive. After six years, I’m moving on.

Working for IBM has been a tremendous experience. Fresh out of college in 2005, I had little idea what I was signing up for. My first task was rather menial: packing the office for its move across town. Immediately after that, Katrina hit and I was tasked with building a quick-and-dirty job search engine to help displaced workers. It was that project that revealed to me the scope and reach of IBM’s influence. No longer was I hacking on dorm room projects; now I was building sites that ended up featured on cable news broadcasts.

In the years that followed, I had the great fortune and privilege to work on a variety of projects supported by IBM’s corporate citizenship group: designing and building the platform that powers the SME Toolkit, a partnership with the International Finance Group and the World Bank; rebuilding the online presence of the State Hermitage Museum in St. Petersburg, Russia; and building tools for American Corporate Partners to help our nation’s veterans.

By far the most exciting experience I had at IBM was the Corporate Service Corps, a 4 week leadership development program in Arusha, Tanzania. Paired with 7 other IBMers, we assisted local NGO and a college with various projects. This was unlike anything I have done: no programming, no Internet. Every day was a new challenge, a new surprise, and we did our best to help our partners with whatever they could throw at us.

What’s next? I’m joining the Obama for America re-election campaign. It’ll be fun.

As noted by our associates at Atomic Object, XML doesn’t allow for nested<![CDATA[…]]> elements. In the course of rewriting some pieces of code, I developed the following Builder workaround to allow our application to export valid XML by breaking the nested CDATA elements into distinct chunks. When read back in via our Nokogiri-based parser, it concatenates the values automagically, and the end result is clean, valid XML.

Fix code:

module Builder
  class XmlMarkup < XmlBase
 
    def cdata_with_escaping!(text)
      if text =~ /(\]\]>)/
        text.gsub!(/(\]\]>)/, "]]]]><![CDATA[>")
      end
      cdata_without_escaping!(text)
    end
    alias_method_chain 'cdata!', 'escaping'
 
  end
end

Sample output:

>> xml = Builder::XmlMarkup.new(str)
>> xml.cdata!("<![CDATA[Foo bar sna]]>")
>> xml.target!
=> "<![CDATA[<![CDATA[Foo bar sna]]]]><![CDATA[>]]>"  # valid XML!
>> xml.cdata_without_escaping!("<![CDATA[Foo bar sna]]>")
>> xml.target!
=> "<![CDATA[<![CDATA[Foo bar sna]]>]]>" # invalid XML!

Sample parsing with Nokogiri:

>> doc = Nokogiri::XML("<baz><![CDATA[<![CDATA[Foo bar sna]]]]><![CDATA[>]]></baz>")
=> #<Nokogiri::XML::Document:0x825aff3c name="document" children=[#<Nokogiri::XML::Element:0x825afc1c name="baz" children=[#<Nokogiri::XML::CDATA:0x825af99c "<![CDATA[Foo bar sna]]>">]>]>
>> doc.css('baz').first.content
=> "<![CDATA[Foo bar sna]]>"

Today I wanted to graph the number of registrations recorded by a Drupal site grouped by date. Drupal stores all account data in the users table. To identify accounts that are registered and verified, I queried with “login != 0″ in my WHERE clause, e.g.

SELECT COUNT(*) FROM user WHERE login != 0;

Since Drupal stores all dates as PHP-style Unix timestamps, e.g. 1284157128, I needed to convert those into a form that MySQL understands. I used from_unixtime() to convert the date to a MySQL date type. By casting, I was then able to use the column in a GROUP BY clause, yielding my result:

SELECT COUNT(*), DATE(FROM_UNIXTIME(created)) as created_date
FROM users WHERE login != 0 GROUP BY created_date;

Which produced results just as I wanted them:

While working on a project for the State Hermitage Museum last year, I had to implement some image watermarking. The basic requirement was that for a certain type of uploaded image, its largest thumbnail should have the museum’s logo tiled across it. I was using attachment_fu to handle the image upload, and ImageMagick/RMagick to process the thumbnails.

After some cursory Googling, I found the ImageMagick Annotating guide, which had this sample watermark command:

 $ convert overlay.png  -fill grey50 -colorize 40  miff:- |\
    composite -dissolve 15 -tile  -  original.jpg watermarked_image.jpg

The dissection of the command:

overlay.png: The source image to overlay

-fill grey50 -colorize 40: Alter the colors of the watermark file

composite: command to overlay the watermark

-dissolve 15 -tile: “dissolve” the overlay at 15%, for good transparency, and tile (repeat) the watermark over the source image.

That’s simple enough, and with these source files:

overlay.png

and dearest Rufus:

rufus.jpg

 $ convert overlay.png -fill grey50 -colorize 40 miff:- |\
    composite -dissolve 15 -tile - rufus.jpg result-15.jpg

Produces:

Overlay with 15% dissolve

convert overlay.png -fill grey50 -colorize 40 miff:- |\
  composite -dissolve 50 -tile - rufus.jpg result-50.jpg

Overlay with 50% dissolve

Unfortunately, RMagick’s watermark method doesn’t support tiling. To work around, I had to call the composite_tiled! method on a colorized image. This code is in my Thumbnail model, which includes attachment_fu:

class Thumbnail < ActiveRecord::Base
  has_attachment  :content_type => :image,
   # some settings omitted
   :watermark_overlay => File.join(RAILS_ROOT, '/public/images/watermark-overlay-image.png'),
   :watermarkable_size => "1500>" 
 
  after_attachment_saved do |record|
    if record.respond_to?(:parent_id) and record.parent_id.nil? # the original image, not the smaller thumbnails
      with_image record.full_filename do |img|
        img.composite_tiled!(
          Magick::ImageList.new(attachment_options[:watermark_overlay]).first.colorize(0.4, 0.4, 0.4, 'grey'),   # process and colorize image
          Magick::SoftLightCompositeOp)
        img.write record.full_filename  # save image
      end
    end
  end
end

Now every Thumbnail record will automatically have a watermarked large image.

I’m working on a small project that involves loading a UTF-16LE (16-bit Unicode, Little Endian) CSV file, converting it to UTF-8 (normal Unicode, as it may be) with iconv, then parsing the values with FasterCSV. Everything was working fine except for loading the first column of data by the column header value. For example, given data:

I could access column 2 (Last Name) as either row.field("Last Name") or row.field(1). However, if I tried to access the first column using row.field("First Name"), it would return nil. row.field(0), on the other hand, would return the proper value.

Hmmmm.

After some sleuthing, I examined the raw content of the string:

(rdb:1) p row.headers.first.unpack('C*')
[239, 187, 191, 70, 105, 114, 115, 116, 32, 78, 97, 109, 101]

Ah, ha! The first three characters are the byte-order mark, or BOM. Ruby, for whatever reason, does not strip it when reading a file as input, so it’s passed along in the input stream. When loading a file with FasterCSV, it’ll keep those characters in the key name, causing lookups by the first column key name to return nil.

I modified my file conversion code as follows:

  def convert_to_utf8
    # Data files are exported as Little Endian UTF-16. We need to parse as UTF-8
    contents = File.open(@file_name).read      
    begin
      converted = Iconv.iconv('UTF-8', 'UTF-16LE', contents)
      converted.first.gsub!("\xEF\xBB\xBF", '') # strip the BOM (byte order mark) from the first line of input
      output = File.open(@file_name, 'w')
      output.write(converted)
    rescue Iconv::Failure
      puts $!.inspect
    end
  end

And all is well in the world.

Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing).

– rsync man page

Ow, my head hurts.

A handy Bash script to display a tree view of a directory, adapted from http://www.centerkey.com/tree. This version omits .svn and .git directories, and uses the find utility.

echo
if [ "$1" != "" ]  #if parameter exists, use as base folder
   then cd "$1"
   fi
pwd
find . \! \( -path "*.svn*" -or -path "*.git*" \) -type d | \
   sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'
if [ `ls -F -1 | grep "/" | wc -l` = 0 ]
   then echo "   -> no sub-directories"
   fi
echo
exit

Example use:

[cgansen@Crystal-Frontier ~]$ tree projects/self.d-struct.org/wp-admin/
 
/Users/cgansen/projects/self.d-struct.org/wp-admin
   .
   |-css
   |-images
   |-import
   |-includes
   |-js
 
[cgansen@Crystal-Frontier ~]$

I’m pleased to announce the relaunch of Jobs4Recovery.com. I programmed the first version of this site in September 2005, in the aftermath of the Katrina-Rita one-two hurricane punch. After the initial wave of activity, the site fell by the wayside. The US Chamber of Commerce, in partnership with IBM, is ressurecting the site to deal with both economic events and natural disasters. Over the past few weeks I’ve worked on upgrading and refreshing the site. I’m pleased with how it turned out, and I hope it’ll help folks when they need it the most.

On a technical note, sometimes it’s fun to switch your whole working environment for a while. I’ve been hacking Ruby on Rails for the past few years, but went back to PHP for this project. While I’ve fallen in love with the Ruby language and the Rails framework, the immediacy of PHP is refreshing. To upgrade this site, almost all of the effort was in modifying the Javascript calls to reflect changes in the Google Maps API, or tweaking layout issues in Internet Explorer. The same core PHP code from 3.5 years ago worked flawlessly, without changing a single line.

Flag of Tanzania

I’m heading to Africa in a few days as a part of the IBM Corporate Service Corps. Despite the mildly awkward name, it’s a rather interesting program. IBM is sending 600 of its best and brightest to work with NGO in emerging markets. It’s a mish-mash of corporate citizenship, good public relations, philanthropy, and advance market research. I’m joining 8 other IBMers from around the globe — the Philippines, India, Brazil, Great Britain, Italy, Japan, and the United States — to work for one month with three organizations in Arusha, Tanzania. We have never worked together as a team before, let alone ever met in person. We’re all hail from various backgrounds which run the gamut from technical skills to sales to marketing to management. Some of us don’t speak English very well, and none of us speak any Swahili.

I learned just a few days ago that I’ll be working with the Institute of Accountancy Arusha (IAA). They offer various undergradute and postgraduate programs in business, accounting, and information technology. I’ll be working with them to assess their current infrastructure, plan upgrades, and various other IT-related tasks. Other team members are working with the African Wildlife Foundation and the Tanzanian Association of Tour Operators on projects ranging from AIDS outreach to marketing to business planning. We’re not the only IBM team that has been to Tanzania. Two teams visited in 2008 to work with TATO and AWF. They accomplished a lot, including creating the current TATO web site, numerous business plans, and lots of consulting work. A fourth team is leaving in a month to go work in the capital of Tanzania, Dodoma.

Overall, I am quite excited for this. It is certainly the most interesting task I’ve taken on in my short tenure with IBM, and I have no doubts that it will be the most challenging. It’s a bit intimidating going into a new country, culture, and pace fo life. Things I take for granted here like instant, always-available Internet access, are simply not available there. What will I ever do with out Google? But that’s all part of the experience, and I look forward to it. I am also quite eager to join this new team. Just from the few weeks of conference calls and uncountable email chains, they have demonstarated a great level of capability and profressionalism.