Kaboose Queue (kabqueue)

The Kaboose Queue system is designed to handle asynchronous generic tasks (or messages). The main function of this system is to take the load off the request cycle. We use this system to send out e-mail, perform intensive image or video tasks, asynchronous network operations, etc.

We had a few important things in mind when desiging this system:

• Performance
• Scalibility
• Reliability

The first item is obvious. The system needs to be fast and relatively resource friendly. For this reason, we chose the Starling queue system from Twitter (www.twitter.com). The second item, scalibility, means that the system should:

• Be distributed. Many machines submitting tasks, and many machines processing these tasks.
• Handle a large number of requests. Each photo upload spawns resize operations for frequently used sizes. Each resize spawns off additional S3 uploads, for example.

The third item requires the system to handle failures, retry common errors, and notify us in case of major failures. Also, the system needs to surive the server dying and coming back and temporary network errors.

With the help of Starling, and our previous experience with a database-backed message queue system, I think we‘ve achieved our goals. We look forward to the community‘s feedback.

Requirements

Starling:

  sudo gem install starling
  sudo gem install daemons

Installation

  script/plugin install http://kabqueue.googlecode.com/svn/trunk/kabqueue

Getting Started

We‘ll examine a not-too-complex scenario here to help you get a handle on how this whole thing works. First, let‘s say you want to send out an e-mail to the album owner when someone comments on their album. We can certainly do this as part of the comment user request, but it‘s better to defer it to later. So, let‘s get started:

*Publish a comment task/message* Add this to your controller (or model) that deals with commenting:

  Kaboose::Task.publish :comment, :album_id => album.id, :time => Time.now, :comment => params[:comment]

*Write the processor* Create a file called comment.rb and put in RAILS_ROOT/app/processors/:

  class FooProcessor < Kaboose::Processor
    processes :album

    def process
      Mailer.deliver_comment_mail(:album => album, :comment => options[:comment], :time => options[:time])
    end
  end

Here is what you need to know about this class.

First, you will need to create a processor file(+foo.rb+) in +RAILS_ROOT/app/processors+ folder. It should be structured like so:

  class FooProcessor < Kaboose::Processor

    processes :some_model

    def process
      some_model.some_method
      end
    end
  end

Each processor class must implement a process method that defines the action that needs to be processed by the queue. This method has access to the _@task_ instance variable that is an instance of Kaboose::Task. The processes macro creates an accessor method as a shortcut for accessing ActiveRecord models specified by model_id option in the task. See Kaboose::Processor‘s self.processes for more info.

Configuration

You will need a kqueue.yml file in your apps config folder to specify the address and namespace of the system, for example:

  address: 127.0.0.1:22122
  namespace: some_namespace

Running the Kaboose Queue system

Just run ./script/queue_processor to get a list of options.

If you are using monit, here‘s what worked for us:

  check process queue-processor with pidfile /path/to/queue_processor.pid
    group qtp
    start program = "PATH/queue_processor start -d -e production -c CWD -u USER -g GROUP"
    stop program = "PATH/queue_processor stop -d -e production -c CWD"

Contact Info, Credits

Blog: labs.kaboose.com/

Google Code Page: code.google.com/p/kabqueue/

Google Group: groups.google.com/group/kaboose_labs

By: Mazdak Rezvani. Kaboose Inc. (www.kaboose.com)

Contributors: Ahmed El-Daly, Talha Syed, Narsa Chelluri

Original non-starling version: Terry Gregory, Mazdak Rezvani. BubbleShare (www.bubbleshare.com)