Geeky Code Stuff

I was wanting to let my users execute arbitrary ruby code to give them the ultimate power to create in my little mud project. But I didn’t want them doing dangerous things or wiping my filesystem. I investigated tainting as well as creating a DSL with treetop.

But I came across a cool experimental project on github called safemode. Unfortunatley it wasn’t packaged as a gem – so I grabbed it from git and packaged it as a gem and pushed it to rubygems. It’s pretty simple to use:

require 'rubygems'
require 'erb'
require 'safemode'

erb_code = %q{<% 10.times do |i| %><%= i %><% end %>}

raw_code = %q{
  (1..10).to_a.collect do |i|
    puts i
    i * 2
  end.join(', ')
}

box = Safemode::Box.new

puts "Doing the ERB code in safe mode\n-----" 
puts box.eval(ERB.new(erb_code).src)

puts "\nDoing the regular Ruby code in safe mode\n-----" 
puts box.eval(raw_code)

puts "\nOutput from regular Ruby code\n-----" 
puts box.output

I have been working on a little side project which is a MUD (multi user dungeon) – mostly because I really love the text based adventure game concept. For me it’s a bit like reading a book compared to watching a movie. The overall feeling is more immersive and lets the imagination play a greater role in the experience – for me at least.

One of the key areas in a MUD is dealing with textual input from the player and then parsing that for the commands the player wants to perform. There is a well known mud project that has been going on for many years in the ruby world called faeriemud – which several people seem to have been working on bit by bit and Michael Granger one of the main developers must have also come across the need to parse and make sense of text from a player. Which is why he has created ruby linguistics – which basically is a set of string functions that can do things like find plurals from non plurals and many other things.

One of the things it does that I was interested in – is 2 different plugin modules for linguistics.

• link parser
• wordnet

so the link parser makes it possible to parse a sentence and find the subject, verb and object and other cool stuff. Wordnet is a database a bit like a dictionary and thesaurus – I had never heard of it before but it seems pretty cool. So I decided to get this up and running. It was a bit of a hunt to get everything I needed and for that reason I have detailed my steps to install it on my my OSX Lion mac pro.

Step 1

Download all the required packages:

• ruby linkparser – wget http://deveiate.org/code/linkparser-1.1.0.gem
• link grammar – wget http://www.abisource.com/downloads/link-grammar/4.7.4/link-grammar-4.7.4.tar.gz
• ruby wordnet – wget http://deveiate.org/code/wordnet-0.0.5.gem
• ruby wordet zip – wget http://deveiate.org/code/wordnet-0.0.5.tar.gz
• wordnet – wget http://wordnetcode.princeton.edu/3.0/WordNet-3.0.tar.gz
• ruby linguistics – sudo gem install linguistics
• berkleydb – download berkleydb from http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html
• ruby bdb – https://github.com/knu/ruby-bdb

Step 2

I put all those downloaded packages into a single directory and did the following:

Build wordnet

tar -xvf WordNet-3.0.tar.gz
cd WordNet-3.0
./configure && make && sudo make install

Build berkleydb

tar -xvf db-5.2.36.tar.gz
cd db-5.2.36/build_unix
../dist/configure LDFLAGS='-arch x86_64 -arch i386' CFLAGS='-arch x86_64 -arch i386'
make && sudo make install

Build ruby bdb

git clone https://github.com/knu/ruby-bdb.git
cd ruby-bdb
ruby extconf.rb --with-db-dir=/usr/local/BerkeleyDB.5.2
make
sudo make install

Build Link Grammar

tar xvf link-grammar-4.7.4.tar.gz 
cd link-grammar-4.7.4
./configure && make && sudo make install

Install gems

sudo gem install linguistics
sudo gem install linkparser-1.1.0.gem
sudo gem install wordnet-0.0.5.gem

Convert wordnet database

tar -xvf wordnet-0.0.5.tar.gz
cd wordnet-0.0.5
ruby convertdb.rb 

The convert db script asks a couple of questions which I just hit enter to use the default and it converts the wordnet data files. Finally I needed to copy the ruby-wordnet files from the wordnet-0.0.5 dir to the location the gem expects to find them:

sudo cp -R ruby-wordnet /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/share
sudo chmod -R 755 /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/share/ruby-wordnet

Step 3

Now to test it all works: (refer to the docs here: http://deveiate.org/projects/Linguistics/wiki/English)

irb
>> require 'linguistics'
=> true
>> Linguistics::use( :en ) 
=> [String, Numeric, Array]
>> "box".en.plural
=> "boxes" 
>> Linguistics::EN.has_wordnet?
=> true
>> "balance".en.synset( :verb )
=> #<WordNet::Synset:0x10f441fa0/2673134 balance, equilibrate, equilibrize, equilibrise (verb): "bring into balance or equilibrium; "She has to balance work and her domestic duties"; "balance the two weights"" (verb_groups: 2, hypernyms: 1, hyponyms: 5, derivations: 7, antonyms: 1)>
>> Linguistics::EN.has_link_parser?
=> true
>> "he is a big dog".en.sentence.object.to_s
link-grammar: Info: Dictionary found at /usr/local/share/link-grammar/en/4.0.dict
=> "dog" 

I came across a need to see the output of a bash shell command in realtime. Usually when you use the backticks or system command you only see the output after the process has run. So in order to get realtime feedback I came up with this:

require 'pty'

def trace_output_for_command(command)
  begin
    PTY.spawn( command ) do |stdin, stdout, pid|
      begin
        stdin.each { |line| print line }
      rescue Errno::EIO
        puts "Output Finished" 
      end
    end
  rescue PTY::ChildExited
    puts "The child process exited!" 
  end
end

trace_output_for_command('ping -c 3 localhost')

When running with MRI / standard ruby this seems to work fine. But when running with jruby instead of exiting after the command has executed it just hung – so I made a small adjustment which fixed that:

When using Jruby:

require 'pty'

def trace_output_for_command(command)
  begin
    PTY.spawn( command ) do |stdin, stdout, pid|
      begin
        stdin.each do |line| 
          print line
          break if stdin.eof?
        end 
      rescue Errno::EIO
        puts "Output Finished" 
      end
    end
  rescue PTY::ChildExited
    puts "The child process exited!" 
  end
end

trace_output_for_command('ping -c 3 localhost')

Finally here is another version which works on unix and returns the exit code of the last process that ran in the shell. Usually I would use $?.exitstatus but it doesn’t seem to be available when using PTY.spawn.

require 'pty'

class ShellCommand

  attr_reader :return_value

  def initialize(command)
    @command = command
    @return_value = 0
  end

  def run_with_trace
    begin
      PTY.spawn( @command + "; echo ${PIPESTATUS[0]}") do |stdin, stdout, pid|
        begin
          stdin.each do |line| 
            puts line
            if stdin.eof?
              @return_value = line.to_i
              break
            end
          end
        rescue Errno::EIO
          puts "Output Finished" 
        end
      end
    rescue PTY::ChildExited
      puts "The child process exited!" 
    end
  end

  def ran_successfully?
    @return_value == 0
  end

end

shell = ShellCommand.new("la -al")
shell.run_with_trace
p shell.ran_successfully?
p shell.return_value

puts "hello this is cool" 

A couple of interesting projects which are related to forking and such:

• Spoon: http://gist.github.com/321084
• Popen4 : http://popen4.rubyforge.org/

both installable as gems: gem install spoon open4

I came across some ways of specifying different methods for everyday things and how to specify your own delimiters in some of these:

Different ways to enclose a multi-line string

data<<-EOF
 Some multiline
 text here
EOF

data = " 
 Some multiline
 text here
" 

%q{
 Some multiline
 text here
}

# With %q you can specify any operator to act as a delimiter
%q| some text |
%q+ some text +

There are a few variations on the %q theme:

• %q is equivalent to a string with single quotes
• %Q is equivalent to a string with double quotes
• %x is equivalent to using the backticks

Another similar function is:

• %w{apple computers are sexy} – which produces a string array of the words provided e.g. [“apple”,”computers”,”are”,”sexy”] – you can use your own delimiters in the same way as %q above.

Recently at work I needed to synchronise a number of threads running in the ruby based automated test tool that we have written. To test for any concurrency issues in our application we wanted to have the threads running each of our test scenarios to collect and wait until they had all reached a specific point and then carry on. This ensures that for every test we can execute business process transactions concurrently.

I borrowed the solution from Java – where they implement a countdown latch. Essentially there is a count and each thread that reaches the designated area causes the count to decrement until it reaches zero and all the threads are released.

I could have gone for a cyclic barrier but that was a bit more complicated than I needed. So to understand how I implemented the countdown latch lets set up some threading. A simple thread pool which consists of worker threads. We can then add jobs to the thread pool and let it handle creating worker threads (or re-using ones that are free) to do the actual work.

Here is the thread pool including the countdown latch:

require 'thread'
require File.expand_path(File.dirname(__FILE__) + '/worker_thread')

class CountDownLatch

   def initialize(count)
     @count = count
     @mutex = Mutex.new
     @conditional = ConditionVariable.new
   end

   def countdown_and_await
      @mutex.synchronize do
       @count -= 1
       while @count > 0
         p "waiting in latch" 
         @conditional.wait(@mutex)        
       end
       @conditional.broadcast
     end
   end

end

class ThreadPool

  attr_accessor :maximum_threads, :timeout
  attr_reader :worker_threads

  def initialize(settings={})
    settings = default_settings.merge(settings)
    @maximum_threads, @timeout = settings[:maximum_threads], settings[:timeout]
    @worker_threads = []
    @mutex = Mutex.new
  end

  def add_job(description=nil, &block)
   assign_job(description,block)
  end

  def add_synchronised_job(description,block)
     assign_job(description,block)
  end

  def wait_until_finished
    start_time = Time.now
    while busy?
      raise_ruckus if (Time.now - start_time) > @timeout
      allow_time_to_process
    end
  end

  def thread_pool_busy
    busy?
  end

  private

  def assign_job(description,block)
     while true
        @mutex.synchronize do
          worker_thread = worker_thread_available?
         # p worker_thread.inspect
          return worker_thread.set_block(block, description) if worker_thread
        end
        allow_time_to_process
      end
  end

  def default_settings
    {
      :maximum_threads => 10,
      :timeout => 300
    }
  end

  def allow_time_to_process
    sleep 0.01
  end

  def worker_thread_available?
    free_worker_thread? || create_new_worker_thread
  end

  def free_worker_thread?
     @worker_threads.each{|worker_thread| return worker_thread unless worker_thread.busy?};false
  end

  def clean_up_idle_threads
    @worker_threads.each{|worker_thread| worker_thread.exit}
  end

  def create_new_worker_thread
    return nil if @worker_threads.size >= @maximum_threads
    worker_thread = WorkerThread.new
    @worker_threads << worker_thread
    worker_thread
  end

  def busy?
    @mutex.synchronize {@worker_threads.any? {|worker_thread| worker_thread.alive? and worker_thread.busy?}}
  end

  def raise_ruckus
    puts "**** blocking threads: " 
    @worker_threads.each do |t|
      puts "    " + (t.description || "(WorkerThread #{t.__id__})") if t.alive? && t.busy?
    end
    raise "PANIC: thread pool wouldn't exit after #{@timeout} seconds!" 
  end
end

This is the worker thread that is used in the thread pool:

class WorkerThread

  attr_accessor :description

  def initialize
    @mutex = Mutex.new
    @thread = Thread.new {
      while true
        sleep 0.05
        block = get_block
        if block
           block.call
          reset_block
        end
      end
    }
  end

  def get_block
    @mutex.synchronize{@block}
  end

  def set_block(block, description)
    @mutex.synchronize{
      raise WorkerThreadBusy, "Thread already busy." if @block
      @block, @description = block, description
    }
  end

  def reset_block
    @mutex.synchronize {@block = @description = nil}
  end

  def busy?
    @mutex.synchronize {!@block.nil?}
  end

  def exit
    @mutex.synchronize {@thread.exit}
  end

  def alive?
    @mutex.synchronize {@thread.alive?}
  end

end

class WorkerThreadBusy < Exception
end

Now we have a basic threading model lets see how we can use the latch:

require 'thread_pool'

thread_pool = ThreadPool.new(:maximum_threads => 2)

def login(item)
  puts "Login #{item}" 
end

def bind_latch(latch,data)
  Proc.new { 
      latch.countdown_and_await
      login(data)
     }

end

login_latch = nil
login_count = 2

(0...100).each_with_index do |item, index|
  data = "data_#{item}" 

  login_latch = CountDownLatch.new(login_count) if  index%login_count==0
  thread_pool.add_synchronised_job("Login #{data}",bind_latch(login_latch,data))

end

thread_pool.wait_until_finished 

We have a loop 0 to 100 and each time we go around the loop we add a job to the thread pool that executes a login function. But we don’t want all of our threads to go off and login instead we want them to synchronize in pairs of 2 threads and then login. So we create a new countdown latch per 2 threads to ensure that we always have 2 threads concurrently running the login function.

If we had more points that we want to synchronise at then we just create some more latches and drop them before the function we want to execute. The login_count above is important to get right – as its the count that the latch counts down from. You always need to have at least 2 threads for the above scenario to work since one thread will wait at the mutex in the latch and another thread is required to reach the same point to decrement the counter and signal the latched threads to be released.

Also because of the way the thread pool is implemented we need to bind the latch and the function to be executed before we add the job to the thread pool otherwise the first thread will wait forever in the latch.

I recently moved to using Snow Leopard on my mac pro and went thru a bit of pain getting rails and mysql to work. It turns out that ruby runs by default in 64 bit mode on snow leopard and so all my previous gems were compiled against a 32 bit ruby. So I had to re-install a list of the gems – luckily I came across this handy ruby script to show me what needed doing:

#!/usr/bin/env ruby 
puts "looking for the gems to upgrade..." 
gem_info = Struct.new(:name, :version)
to_reinstall = []
Dir.glob('/Library/Ruby/Gems/**/*.bundle').map do |path| 
  path =~ /.*1.8\/gems\/(.*)-(.*?)\/.*/
  name, version = $1, $2
  bundle_info = `file path`
  to_reinstall << gem_info.new(name, version) unless bundle_info =~ /bundle x86_64/
end

gemnames = to_reinstall.map{|ginfo| ginfo.name}.uniq.delete_if{|name| name =~ /mysql|passenger/}
puts "***" 
puts "Please reinstall:" 
gemnames.each do |name|
  gems = to_reinstall.select{|ginfo| ginfo.name == name}
  puts "#{name} versions: #{gems.map{|ginfo| ginfo.version}.join(', ')}" 
end 

puts "or uninstall all gems that need to be reinstalled:\n" 
puts "$ sudo gem uninstall #{gemnames.join(' ')}" 
puts " " 
puts "and reinstall them:\n" 
puts "$ sudo gem install #{gemnames.join(' ')}" 

But before I even got to running that script when I was trying to do a rake db:create:all I was getting this annoying error:

Couldn't create database for {"username"=>"my_username", "adapter"=>"mysql", "database"=>"my_database", "password"=>"my_password"}, charset: utf8, collation: utf8_general_ci (if you set the charset manually, make sure you have a matching collation)

So a bit more digging showed me that to avoid weird errors with mysql I should move to using the 64 bit version of mysql. So I uninstalled the 32 bit version by doing this:

sudo rm /usr/local/mysql
sudo rm -rf /usr/local/mysql*
sudo rm -rf /Library/StartupItems/MySQLCOM
sudo rm -rf /Library/PreferencePanes/My*
edit /etc/hostconfig and remove the line MYSQLCOM=-YES-
rm -rf ~/Library/PreferencePanes/My*
sudo rm -rf /Library/Receipts/mysql*
sudo rm -rf /Library/Receipts/MySQL*

and then installed the 64 bit version and uninstalled the mysql gem and then installed it again:

sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Thanks to a few ruby libraries it’s pretty simple to create a testing tool using ruby that can send post and get requests to a webserver and then parse the result. The main benefit of using this kind of test approach is when you want to run a suite of tests in a continuous integration build and get fast feedback if something has changed or broken.

Some people prefer to go for tools that fire up a browser and then drive the browser – tools like:

• Selenium
• Watir

There are many others but the main downfall to this type of tool is that it is often really slow, especially when people start churning out a lot of tests. Some people go for a combined approach – they run post/get type tests in the build on a day to day basis and then run the same tests but with the browser once or twice a day or overnight – this approach seems quite common with webdriver the java based tool. It lets you choose how you want to run your tests – headless or with a browser or another runner of your choice.

So the approach I’m going to take here is to create a DSL (Domain Specific Language) type of test suite. By DSL I mean it will allow you to write tests using language that is appropriate to the domain we are working with. This is going to be really simple and we are going to be writing tests for Google.

Lets install some gems we will need:

• sudo gem install mechanize rspec syntax

Now lets write some code. Make a directory called GoogleTest and inside create the file google.rb

class Google

  FIELDS = {
          :search_field => 'q',
          :location => 'meta'
  }

  def initialize(agent,page=nil)
    @page = page
    @agent = agent
    @form = @page.forms.first unless @page.nil?
  end

  def navigate
    @page = @agent.get("http://www.google.co.uk")
    @form = @page.forms.first
  end

  def search_term=(value)
    set_textfield(:search_field, value)
  end

  def search_term
    get_textfield(:search_field)
  end

  def location=(value)
    set_radiobutton(:location,value)
  end

  def location
    get_radiobutton(:location)
  end

  def google_search
    button_with_value("Google Search")
  end

  private

  def set_textfield(field_name, value)
    @form.field_with(:name => FIELDS[field_name]).value = value
  end

  def get_textfield(field_name)
    @form.field_with(:name => FIELDS[field_name]).value
  end

  def set_radiobutton(field_name, value)
    @form.radiobutton_with(:name => FIELDS[field_name]).value = value
  end

  def get_radiobutton(field_name)
    @form.radiobutton_with(:name => FIELDS[field_name]).value
  end

  def button_with_value(value)
    button = @form.button_with(:value => value)
    @page = @agent.submit(@form, button)
  end

end

As you can see here we are mapping out the fields available on the google search form with appropriate methods. This will help us to write rspec specifications more relevant to the domain of google. So now lets create a spec file in the GoogleTest directory – google_search_spec.rb

require 'rubygems'
require 'mechanize'
require File.expand_path(File.dirname(__FILE__) + '/google.rb')

describe Google do

  before(:all) do
    @agent = WWW::Mechanize.new
    @agent.read_timeout = 240
  end

  before(:each) do
    @google = Google.new(@agent)
    @google.navigate
    @search_term = "software testing" 
  end

  it "should perform a search with the supplied search term" do
    @google.search_term = @search_term
    result = @google.google_search
    number_of_matching_links(result,@search_term).should have_at_least(20).items
  end

  it "should perform a search with pages from the uk" do
    @google.search_term = @search_term
    @google.location = "cr=countryUK|countryGB" 
    result = @google.google_search
    number_of_uk_links(result).should have_at_least(7).items
  end

  def number_of_matching_links(result,term)
    result.search("a").collect{|link| link if link.inner_text.match(/#{term}/i)}.compact
  end

  def number_of_uk_links(result)
    result.search("a").collect{|link|
      link if link.attributes['href'].text.match(/\.co\.uk/) and !link.attributes['href'].text.match(/google/)}.compact
  end

end

Rspec lets you describe the general area you want to test – in this case its our Google class. Next we setup variables that we will use throughout the tests. We instantiate a new mechanize object that has all the http connection stuff. I have set the timeout just to show how its done – it’s not really needed in this example.

Next we setup variables that we want to use in each of the tests that get instantiated freshly each test – again this was not strictly required for this example but I wanted to show you this feature of rspec for future reference.

Now we get down to some testing. We set the search term on the google object and then run the google search and check that the number of links returned is at least 20. In the second test we do the same but we set the location radio button to use pages from the uk only (If you are in a different country you might need to tweak the urls and values a little). Finally we check that the result contains at least 7 links that have co.uk in them.

The nice thing about this approach is that we can build up some common language about how to perform google searches and if google decides to change the structure of its webpages we can keep the test exactly the same but change the supporting classes behind.

Lets run the tests and see the results on the command line:

• spec google_search_spec.rb

and you should see:

You can also run a test on a specific line number which is quite useful:

• spec -l18 google_search_spec.rb

And for running in a continuous integration build you can get it to print an htlm report:

• spec -f html:./result.html -f p google_search_spec.rb (the p just gives cmd output as well)

Lets make it fail just to prove it’s actually working. Edit the spec and change the expected result to 300:

  it "should perform a search with the supplied search term" do
    @google.search_term = @search_term
    result = @google.google_search
    number_of_matching_links(result,@search_term).should have_at_least(300).items
  end

Run it again and see the result:

This approach is great for fast feedback but it does have a few shortcomings when dealing with ajax based websites. Handling ajax post backs isn’t too difficult as we can just submit the form again to get the updated call back data. Trying to test pages that have alot of dynamic javascript is much much harder. A good example on the google search page is the I’m Feeling Lucky button – it uses javascript to take you straight to the first search result in the list. A workaround for this case is to submit a get request to the server something like this:

http://www.google.co.uk/search?hl=en&q=software testing&btnI=I%27m+Feeling+Lucky&meta=&aq=f&oq=

Obviously this was a really simple example to give you the general idea. I have implemented a much more complex suite of tests based on this approach at a client where we have over 300 tests of this nature running on every svn checkin in a cruise build pipeline. The tests take around 30 minutes to run and they run multithreaded.

I have been using a ruby based functional testing tool that I wrote to test a clients web application. The web application uses different currency symbols of which most were quite happily handled by default within ruby.

However the Euro symbol was a bit more of a problem. Just pasting in the € symbol didn’t work out so instead I had to dig around for some help and I found this useful ruby-forum topic where someone had a similar problem: Euro currency symbol

Right at the bottom I found what I was looking for. A way to replace the euro symbol in my string with some ruby code that could generate the right symbol:

   # The string I wanted to use to set a drop
   # down select option on a form
   "HUB  € DAH Argus (Mid)"  

   # The solution
   euro_symbol = [8364].pack("U")
   "HUB #{euro_symbol} DAH Argus (Mid)" 

Today I was trying to get mechanize work with NTLM, but I found that NTLM authentication is different from basic authentication and in the mechanize code there is a decision whether to use basic or somethings else.

so I decided to fix this and I grabbed the rubyntlm gem and edited the mechanize code to use rubyntlm stuff. However I also found that when trying to do this I had to patch the rubyntlm code in order to make using it from net/http easier.

For convenience and because I am using this code where I work on various different machines I have packaged up the ntlm and mechanize code into a gem called mechanize-ntlm. If you just want to use the ruby ntlm-http stuff by itself standalone I packaged that as ntlm-http.

So now in mechanize you can do this:

require 'rubygems'
require 'mechanize-ntlm'

agent = WWW::Mechanize.new

agent.basic_auth("username","password")

page = agent.get('http://some/url')

page.links.each do |link|
   puts link.text
end

and it will correctly realize that you want to use NTLM authentication and work just fine.

Get the mechanize-ntlm gem here: mechanize-ntlm-0.9.1.gem

Get the ntlm-http gem here: ntlm-http-0.1.1.gem

The ntlm-http stuff just helps if you want to do ntlm from the net/http world – for example consider the following:

        when :ntlm;  request.ntlm_auth('username', 'password')
        when :basic; request.basic_auth('username', 'password')

You now are able to call ntlm_auth on the Net::HTTP request object as well as the already existing basic_auth.

There will be some ruby code action coming up very soon:

    class AwesomeRuby

        def initialize(awesomeness)
            puts "Ruby is Awesome!" 
        end

    end