It's a trap!

Using POSIX signals in Ruby is pretty easy. Use the trap method and pass it a block that will execute when the trap is received.

trap("TERM") { puts "TERM signal received."}
sleep(50)

If you run this and send it a TERM signal (using kill <pid> for example), it will print "TERM signal received." One thing to note, this will override Ruby's built in handlers. So, kill will no longer cause the program to exit.

trap will return a reference to the current trap handler. You can use this to replace the previous handler when you are finished.

previous_handler = trap("TERM") { puts "TERM signal received."}

and to restore the previous handler:

trap("TERM", previous_handler)

One poorly documented gotcha is that trap handlers always run in the main thread and not in the thread that the trap was defined. This can cause hard to track down bugs.

On a previous project we were running into weird issues where open database transactions were being committed when the process received a TERM signal. We were using an old version of Rails 1.2 and tracked it down to the transaction method in ActiveRecord::Transactions (condensed to show relevant code):

previous_handler = trap('TERM') { raise TransactionError, "Transaction aborted" }

begin
  ...
  
  result = connection.transaction(Thread.current['start_db_transaction'], &block)
  
    objects.each { |o| o.commit_transaction }
    return result
  rescue Exception => object_transaction_rollback
    objects.each { |o| o.abort_transaction }
    raise
  ensure
    ...
end

This is supposed to raise a TransactionError if a TERM is received, but if Rails is not running on the main thread (which it isn't if you are using Mongrel), the raise will not occur in the transaction thread. So, the rescue will not be executed.

We found a workaround by changing the trap line to this:

transaction_thread = Thread.current
previous_handler = trap('TERM') { transaction_thread.raise TransactionError, "Transaction aborted" }

This will cause the exception to always be raised in transaction_thread.

This doesn't seem to be a problem on the newer version of Rails, but its something to watch out for when using trap.

Synchronizing using Active Record

Recently, I was working on implementing a simple job queue. I needed multiple processes to pull jobs from the queue, and I needed to synchronize access to the job queue. ActiveRecord allows you to pass the :lock option to #find.

def find_new_job
  Job.transaction do
    job = Job.find(:first, :conditions => "status = 'new'", :lock => true)
    if job
      job.status = "processing"
      job.save!
    end
    job
  end
end

ActiveRecord will use a SELECT ... FOR UPDATE, which locks the rows in the result set. When another query selects any of those rows, it will block until the first transaction is complete. This works well on Postgres and MySQL 5, but Oracle gave me an error:

ActiveRecord::StatementInvalid: OCIError: ORA-02014: cannot select FOR UPDATE from view with DISTINCT, GROUP BY, etc.: select * from (select raw_sql_.*, rownum raw_rnum_ from (SELECT * FROM jobs WHERE (status = 'new') ) raw_sql_ where rownum <= 1) where raw_rnum_ > 0 FOR UPDATE
 from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:128:in `log'
 from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/connection_adapters/oracle_adapter.rb:222:in
`execute'
etc...

The Oracle adapter generates the above SQL for a find(:first), and SELECT .. FOR UPDATE won't work on a subselect. So, changing the method to something like the following will allow it to work in Oracle:

def find_new_job
  Job.transaction do
    jobs = Job.find(:all, :conditions => "status = 'new'", :lock => true)
    job = nil
    if jobs.size > 0
      job = jobs.first
      job.status = "processing"
      job.save!
    end
    job
  end
end

I've used this sort of job queue in Rails apps to allow for asynchronous calls to external services. Instead of calling the services synchronously and making the user wait (and use up a mongrel instance), create an entry in the job queue. External ruby processes can poll the job queue looking for work to do. If the user needs to see the results of the job (such as placing an order or authorizing a credit card), the request that created the job redirects to a controller action that polls the status of the job. When the job is completed, the action will redirect to a new action that will display the results of the job.

The same database locking mechanism can also be used as a simple way to synchronize access to resources that are shared across multiple processes on multiple servers. Create a simple ActiveRecord model for a Lock object:

class Lock < ActiveRecord::Base
  def self.synchronize(name)
    self.transaction do
      Lock.find(:first, :conditions => ["name = ?", name], :lock => true)
      yield
    end
  end
end

Create a lock:

Lock.create!(:name => "TestLock")

Then use it like this:

Lock.synchronize("TestLock") do
  # access resource here
end

The downside to this is that all locks you want to use need to be created beforehand. If there are a few predetermined locks needed for the application, this approach works fine.

Photo: Gatun locks at the Panama Canal

Image borrowed from Flickr user: Donna62