Vendoring Binaires on Heroku: An Example with Aspell

Recently I needed to install the Aspell library on Heroku for a side project. By default a Heroku Dyno doesn't include any non-essential libraries so your left to install these on your own. This introduced me to the world of buildpacks and an awesome build tool that Heroku provides to get the job done.

Objective

The objective is to have a Rails application running with FFI::Aspell, a FFI binding for the Aspell library on Heroku.

Heroku Buildpacks and Vendoring Binaries

First, we need to configure Heroku to vendor binaries. The easiest way to do this is use the Vendor Binaries buildpack which allows you to extract a tarball stored on S3 into your /app directory when the application build process is triggered. This is all configured in a .vendor_urls located in the root of your application.

As Heroku only supports a single buildpack by default, we need to configure it to support multiple buildpacks. In our case, we need the Ruby buildpack in addition to the Vendor Binaries buildpack. Fortunately this is relatively easy to do when we create our new application (and can be done after the fact as well):

heroku create --stack cedar --buildpack https://github.com/dollar/heroku-buildpack-multi.git

Now we have our multi buildpack application, we can configure those buildpacks in .buildpacks:

https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
https://github.com/heroku/heroku-buildpack-ruby.git

For our vendored binaries, although we haven't built it yet, we can configure it's location on S3 in .vendor_urls:

http://your-bucket.s3.amazonaws.com/aspell-0.60.6.1.tar.gz

Commit both the .buildpacks and the .vendor_urls files to your Rails application.

Build the binary

Next we need to build the binary. First, let's download and extract the source into a temporary working directory (not in your Rails application).

mkdir ~/Code/temp && cd ~/Code/temp
curl -o aspell-0.60.6.1.tar.gz ftp://ftp.gnu.org/gnu/aspell/aspell-0.60.6.1.tar.gz
tar -xvzf aspell-0.60.6.1.tar.gz

Now, let's create a build server on Heroku. The Vulcan gem automates the process of creating the server and building the library.

gem install vulcan
vulcan create vulcan-yourname

Now we're ready to build the binary:

vulcan build -s ~/Code/temp/aspell-0.60.6.1 -p /tmp/aspell -c "./configure --prefix=/tmp/aspell && make install"

You will notice that we configure the library to be installed in /tmp/aspell using the prefix option. We also need to tell Vulcan using the -p option where the compiled library will be located. At the completion of the build, a tarball is then download to your /tmp directory.

You can now copy the tarball from /tmp/aspell-0.60.6.tgz to the S3 bucket specified in .vendor_urls. Ensure the read permission on the uploaded file can viewed by "world", otherwise it won't be accessible when you deploy your application and you will receive the following errors:

-----> Found a .vendor_urls file
       Vendoring http://yourbucket-heroku.s3.amazonaws.com/aspell-0.60.6.tgz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Exiting with failure status due to previous errors

 !     Push rejected, failed to compile Multipack app

Build the supporting dictionary files

Typically we would be done at this point, but we need to compile the dictionary files that Aspell uses separately. This makes the process a little more complicated.

We're going to use our Vulcan build server to compile the dictionary files. This requires starting up a shell, downloading the Aspell tarbar we built previously and extracting it into the original installation directory.

heroku run bash --app vulcan-yourname
cd /tmp
mkdir aspell && cd aspell
curl -o aspell-0.60.6.tgz http://yourbucket-heroku.s3.amazonaws.com/aspell-0.60.6.tgz
tar -xzvf aspell-0.60.6.tgz

Now we're ready to build the dictionary files:

export PATH=$PATH:/tmp/aspell/bin
curl -o aspell6-en-7.1-0.tar.bz2 ftp://ftp.gnu.org/gnu/aspell/dict/en/aspell6-en-7.1-0.tar.bz2
tar -xvf aspell6-en-7.1-0.tar.bz2
cd aspell6-en-7.1-0
./configure && make install

At this point you will see output similar to this:

/tmp/aspell/bin/prezip-bin -d < en-common.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-common.rws
/tmp/aspell/bin/prezip-bin -d < en-variant_0.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-variant_0.rws
/tmp/aspell/bin/prezip-bin -d < en-variant_1.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-variant_1.rws
/tmp/aspell/bin/prezip-bin -d < en-variant_2.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-variant_2.rws
...

Once the build is completed you can test Aspell is working correctly with:

echo "helloz" | aspell -a

We're almost there. Now just copy the dictionary files over to our original build for Aspell in /lib/aspell-0.60/ We will create a new tarball which includes the dictionary files:

cp *.rws *.alias *.multi *.dat ../lib/aspell-0.60/.
cd ..
tar -czvf aspell-0.60.6.tgz bin include lib share

Finally, transfer the new tarball aspell-0.60.6.tgz to your local machine and upload to S3 again replacing our original tarball.

Deployment

Before deploying you will need to configure LD_LIBRARY_PATH, the path used by the dynamic loader to load libraries into dynamically linked executables. Otherwise, the following error is raised:

aspell: error while loading shared libraries: libaspell.so.15: cannot open shared object file: No such file or directory

This is easy to do:

heroku config:add LD_LIBRARY_PATH=/app/lib

Now let's deploy our application. Remember to include the FFI::Aspell in our Gemfile and bundle. As long as you have your application setup correctly with Heroku you can then:

git push heroku

On a successful build your output will begin with the following (note the vendoring of http://your-bucket.s3.amazonaws.com/aspell-0.60.6.tgz):

-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
=====> Detected Framework: VendorBinaries
-----> Found a .vendor_urls file
       Vendoring http://your-bucket.s3.amazonaws.com/aspell-0.60.6.tgz
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-ruby.git
=====> Detected Framework: Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using Bundler version 1.3.2
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
       Fetching gem metadata from https://rubygems.org/..........
       Fetching gem metadata from https://rubygems.org/..
       Installing rake (10.1.0)

Test the FFI::Aspell Gem

The moment of truth! Let's run a Rails console to test the FFI:Aspell gem.

heroku run console

At the Rails console:

speller = FFI::Aspell::Speller.new('en_US', 'dict-dir' => '/app/lib/aspell-0.60')
speller.correct?('cookie') # => true

Note we need to specify the dict-dir option since the Aspell library looks for it in /tmp/aspell/lib/aspell-0.60 by default (based on the prefix option we used when we compiled it). If you were to execute the binary directly from the shell you would also need to include this option. For example:

echo "helloz" | aspell -a --dict-dir /app/lib/aspell-0.60

If you don't include the dict-dir option, the FFI:Aspell library will crash.

Troubleshooting

If you have problems, it best to try to run the Aspell library directly from the shell using the example command above.

If you receive the following error:

Error: No word lists can be found for the language "en_US".

Then the path to the dictionary files is not correct.

References

• Packaging Binary Buildpack Dependencies
• Vulcan
• Heroku Vendor Binaries Buildpack
• Heroku Multi Buildpack
• Using Hunspell With Heroku

Basic Heroku addons I always use for production websites

heroku addons

cron:daily
custom_domains:basic
logging:expanded
memcache:5mb
newrelic:standard
pgbackups:auto-month
sendgrid:basic
shared-database:5mb

Git: Push a local branch to a remote master

git push heroku yourbranch:master 

Credit Heroku

What Git revision is deployed to Heroku?

Updated: Does not work on Heroku's Cedar stack.

If you want to know which revision has been deployed to Heroku execute the following command in your project:

heroku console "ENV['COMMIT_HASH']"

Credit: http://stackoverflow.com/questions/2281772/view-remote-git-revision-on-heroku/4609925#4609925

Avoid a bloated slug on Heroku by configuring Bundler

This tip was outlined in Heroku's December email newsletter, and wanted to repeat here. It's a must know. You don't need your development and test gems in production, so configure your Heroku application accordingly:

Smaller slug, faster installation and compile times, nice.

Pushing specific tags to Heroku

From the Heroku's October Newsletter, I thought that this was very helpful. Tag a release in Git:

git tag -a v1.1

And push it to Heroku:

git push -f heroku v1.1^{}:master

Installing Redmine on Heroku with S3 Storage: Step by Step

Update: 15 August 2011This blog post was written almost a year ago. I haven't attempted to install Redmine on Heroku since then, so your milage may vary. I do intend to do an updated version in the future as this has been a very popular post.

I've started using Redmine this year, a project management web application, and I'm a big fan. I'm also a big fan of Heroku for deploying Rails applications. So, you know it makes sense -- deploy Redmine on Heroku! Here's how to do it step by step:

Initial Setup

$ mkdir your-company-redmine
$ cd your-company-redmine
$ git init
$ git remote add redmine git://github.com/edavis10/redmine.git
$ git fetch redmine
$ git merge redmine/1.0-stable
$ rake db:create RAILS_ENV=production

# if you don't have these installed
$ (sudo) gem install rails --version 2.3.5
$ (sudo) gem install rack --version 1.0.1

# setup database locally
$ RAILS_ENV=production rake db:migrate
$ RAILS_ENV=production rake redmine:load_default_data

$ rake generate_session_store

# add .gems
rails --version 2.3.5

$ mkdir tmp public/plugin_assets

# remove from .gitignore
/config/initializers/session_store.rb

$ git add .
$ git commit -m "Initial configuration"

Install Plugins

# install plugins, I like to use giternal
$ (sudo) gem install giternal

# add config/giternal.yml
redmine_heroku:
  path: vendor/plugins
  repo: http://github.com/edavis10/redmine_heroku.git
  
redmine_s3:
  path: vendor/plugins
  repo: http://github.com/tigrish/redmine_s3.git

$ giternal update
$ giternal freeze

$ git add .
$ git commit -m "Adds plugins"

Configure Heroku Plugin

# Remove from .gitignore
/public/plugin_assets

$ rake heroku:setup

Configure S3

# config/s3.yml

production:
  access_key_id: YOUR_S3_ACCESS_KEY_ID
  secret_access_key: YOUR_S3_SECRET_ACCESS_KEY
  bucket: YOUR_S3_REDMINE_PRODUCTION_BUCKET
  cname_bucket: false

development:
  access_key_id: YOUR_S3_ACCESS_KEY_ID
  secret_access_key: YOUR_S3_SECRET_ACCESS_KEY
  bucket: YOUR_S3_REDMINE_DEVELOPMENT_BUCKET
  cname_bucket: false

$ git add .
$ git commit -m "Configures S3 plugin"

Configure Sendgrid for Email

# remove from .gitignore
/config/email.yml

$ heroku addons:add sendgrid
$ heroku config

# configure config/email.yml
production:
  delivery_method: :smtp
  smtp_settings:
    address: "smtp.sendgrid.net"
    port: 25
    authentication: :plain
    domain: "heroku.com"
    user_name: "YOUR_SENDGRID_PASSWORD"
    password: "YOUR_SENDGRID_USERNAME"

$ git add .
$ git commit -m "Configures SendGrid for email delivery"

Deploy to Heroku

$ heroku create your-company-redmine
$ git push heroku deploy

# since we have the database setup locally, lets' push it
$ heroku db:push

Get Sass and Compass running on Heroku with Rails 3.0-beta

Heroku is read-only file system causing a little havoc for runtime compilation of Sass files into CSS. In my recent trial of running a Rails 3.0-beta application on Heroku with MRI 1.9.1 I came across this issue.

Doing a google on “heroku sass” will bring you to this post which leads you to sass_on_heroku plugin which is deprecated. The current plugin is Hassle, but unfortunately this hasn’t been touched for a few months. There are some current issues with this plugin and Compass. But all is not lost! Jacques Crocker has forked the plugin and made a fix for this. Get a compatible version of Hassle that works with Compass from his repo.

Now, there’s one change that needs to be made to get Hassle working with Rails 3.0-beta. As Hassle is a piece of Rack middleware, the configuration has changed and must be configured using the Railtie class based on this documentation. Admittedly, this is a little hacky, but gets things going. Placing the Railtie file in it’s own file and namedspace would clean things up.

# init.rb

# Rails 2.3.x style
#
# if RAILS_ENV == 'production'
#   ActionController::Dispatcher.middleware.use Hassle
# end

require 'hassle'

class HassleRailtie < Rails::Railtie
  config.middleware.use Hassle if Rails.env == 'production' 
end    

And, that’s it! With this you should be able to get Sass and Compass running on Heroku with Rails 3.0-beta.

Getting Rails 3.0-beta running on Heroku

I wanted to test out Rails 3.0-beta and see how far I could get with developing an application with my stack of gems/plugins. I hadn’t checked out Heroku either, so it gave me an opportunity to try out their new stacks with Ruby 1.9.1.

First, I installed Rails 3 Beta and RVM outlined in Ryan Bate’s screencast. If you haven’t installed RVM yet, you will be wondering why it’s taken you this long. I did.

Once you have Rails up and running, you can following with their quick start guide. There’s a couple of things to take note when deploying for Rails 3.0-beta, but it’s all pretty straight forward.

As mentioned in the documentation you will need to add the “pg” gem to your Gemfile as Heroku runs Postgres database on their servers; i.e. not MySQL.

  # Gemfile
  group :production do
    gem "pg" 
  end  

By placing it in the “production” group you won’t need to install it locally.

I’m super impressed by Heroku, and look forward to get a production application running on their platform. It’s simply awesome.