Processor Pool plugin

Plugin details

Provides a simple load-balancing solution for Amazon's EC2 - S3 backend.

Usage would include an upload server, a image and video processing and transcoding server, all non-blocking separate from an application server.

Websitehttp://rubyforge.org/projects/processorpool/ Repositoryhttp://processorpool.rubyforge.org/svn Author Ari Lerner Tags amazon, EC2, S3 LicenseMIT

Documentation

Install the plugin:
ruby script/plugin install http://processorpool.rubyforge.org/svn

== SYNOPSIS:

require "rubygems"
require "lib/processor.rb"



# Set the access_key_id and the secret_access_key provided by amazon

access_key_id, secret_access_key = '11H31W1044QZ9QQF48G2', 'a8Xb46qggIeDfm1y/RUL48Ullq87MxgOKCzCmw1c'



# And start the processors!

ProcessorPool.start(access_key_id, secret_access_key)



# ProcessorPool is built on top of Sinatra (http://sinatra.rubyforge.org/) and uses the same idioms.
# For more information, check out Sinatra (at http://sinatra.rubyforge.org/):

get '/' do
"Processing data"
end



== IMPLEMENTATION
# Uploading and processing
# To use it without inline uploading
# Upload like you normally would do, but in the controller, just pass the variables to the offsite storage and continue on as you would. This won't block the rails mongrel from continuing on. Of course, you also need to check to make sure the file does get uploaded and processed at this point.
# Inline uploading
# In Rails, you can redirect the the upload to the processor pool in the form. For instance:

<% form_for @media, :id => "fileform", :url => upload_url, :multipart => true do %>




# For ease of use, we recommend embedding this helper function somewhere:

def url_for_upload_server
if RAILS_ENV == 'production'
  # determine the upload server to try to use
  p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
  "http://#{p.hostname}:4567/new"
else
  p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
  "http://localhost:4567/new"
end
end



# The reset is an exercise for you to complete.

# Other ideas, using similar methodology would be to upload videos and do the processing there, to run awy sort of processor intensive computations off-site.

== CONVENIENCE METHODS

# There are a number of convenience methods included with processor_pool. It is intended to work with Amazon's S3 service.

AWS::S3::S3Object.store('/folder/to/greeting.txt', 'hello world!', 'ron', :use_virtual_directories => true)



# Alternatively, you can use the included RemoteDirectory class

dir = RemoteDirectory.new("/folder/name")



# Save one file to the S3 directory

dir.save_to_s3("movie.mpg", "movies", "encoded")



# Copy all the files in the directory to the s3 bucket

dir.copy_to_s3("movies", "encoded")



dir.files

# Returns a list of all the files in the bucket

Further Documentation

There is currently no advanced documentation for this plugin.

New documentation

Edit plugin | (0 older versions) | Last edited by: Guest, 7 months ago