Blog

Tuesday, October 8, 2013

Vendoring Binaires on Heroku: An Example with Aspell

Recently I needed to install the Aspell library on Heroku for a side project. By default a Heroku Dyno doesn't include any non-essential libraries so your left to install these on your own. This introduced me to the world of buildpacks and an awesome build tool that Heroku provides to get the job done.

Objective

The objective is to have a Rails application running with FFI::Aspell, a FFI binding for the Aspell library on Heroku.

Heroku Buildpacks and Vendoring Binaries

First, we need to configure Heroku to vendor binaries. The easiest way to do this is use the Vendor Binaries buildpack which allows you to extract a tarball stored on S3 into your /app directory when the application build process is triggered. This is all configured in a .vendor_urls located in the root of your application.

As Heroku only supports a single buildpack by default, we need to configure it to support multiple buildpacks. In our case, we need the Ruby buildpack in addition to the Vendor Binaries buildpack. Fortunately this is relatively easy to do when we create our new application (and can be done after the fact as well):

heroku create --stack cedar --buildpack https://github.com/dollar/heroku-buildpack-multi.git

Now we have our multi buildpack application, we can configure those buildpacks in .buildpacks:

https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
https://github.com/heroku/heroku-buildpack-ruby.git

For our vendored binaries, although we haven't built it yet, we can configure it's location on S3 in .vendor_urls:

http://your-bucket.s3.amazonaws.com/aspell-0.60.6.1.tar.gz

Commit both the .buildpacks and the .vendor_urls files to your Rails application.

Build the binary

Next we need to build the binary. First, let's download and extract the source into a temporary working directory (not in your Rails application).

mkdir ~/Code/temp && cd ~/Code/temp
curl -o aspell-0.60.6.1.tar.gz ftp://ftp.gnu.org/gnu/aspell/aspell-0.60.6.1.tar.gz
tar -xvzf aspell-0.60.6.1.tar.gz

Now, let's create a build server on Heroku. The Vulcan gem automates the process of creating the server and building the library.

gem install vulcan
vulcan create vulcan-yourname

Now we're ready to build the binary:

vulcan build -s ~/Code/temp/aspell-0.60.6.1 -p /tmp/aspell -c "./configure --prefix=/tmp/aspell && make install"

You will notice that we configure the library to be installed in /tmp/aspell using the prefix option. We also need to tell Vulcan using the -p option where the compiled library will be located. At the completion of the build, a tarball is then download to your /tmp directory.

You can now copy the tarball from /tmp/aspell-0.60.6.tgz to the S3 bucket specified in .vendor_urls. Ensure the read permission on the uploaded file can viewed by "world", otherwise it won't be accessible when you deploy your application and you will receive the following errors:

-----> Found a .vendor_urls file
       Vendoring http://yourbucket-heroku.s3.amazonaws.com/aspell-0.60.6.tgz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Exiting with failure status due to previous errors

 !     Push rejected, failed to compile Multipack app

Build the supporting dictionary files

Typically we would be done at this point, but we need to compile the dictionary files that Aspell uses separately. This makes the process a little more complicated.

We're going to use our Vulcan build server to compile the dictionary files. This requires starting up a shell, downloading the Aspell tarbar we built previously and extracting it into the original installation directory.

heroku run bash --app vulcan-yourname
cd /tmp
mkdir aspell && cd aspell
curl -o aspell-0.60.6.tgz http://yourbucket-heroku.s3.amazonaws.com/aspell-0.60.6.tgz
tar -xzvf aspell-0.60.6.tgz

Now we're ready to build the dictionary files:

export PATH=$PATH:/tmp/aspell/bin
curl -o aspell6-en-7.1-0.tar.bz2 ftp://ftp.gnu.org/gnu/aspell/dict/en/aspell6-en-7.1-0.tar.bz2
tar -xvf aspell6-en-7.1-0.tar.bz2
cd aspell6-en-7.1-0
./configure && make install

At this point you will see output similar to this:

/tmp/aspell/bin/prezip-bin -d < en-common.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-common.rws
/tmp/aspell/bin/prezip-bin -d < en-variant_0.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-variant_0.rws
/tmp/aspell/bin/prezip-bin -d < en-variant_1.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-variant_1.rws
/tmp/aspell/bin/prezip-bin -d < en-variant_2.cwl | /tmp/aspell/bin/aspell  --lang=en create master ./en-variant_2.rws
...

Once the build is completed you can test Aspell is working correctly with:

echo "helloz" | aspell -a

We're almost there. Now just copy the dictionary files over to our original build for Aspell in /lib/aspell-0.60/ We will create a new tarball which includes the dictionary files:

cp *.rws *.alias *.multi *.dat ../lib/aspell-0.60/.
cd ..
tar -czvf aspell-0.60.6.tgz bin include lib share

Finally, transfer the new tarball aspell-0.60.6.tgz to your local machine and upload to S3 again replacing our original tarball.

Deployment

Before deploying you will need to configure LD_LIBRARY_PATH, the path used by the dynamic loader to load libraries into dynamically linked executables. Otherwise, the following error is raised:

aspell: error while loading shared libraries: libaspell.so.15: cannot open shared object file: No such file or directory

This is easy to do:

heroku config:add LD_LIBRARY_PATH=/app/lib

Now let's deploy our application. Remember to include the FFI::Aspell in our Gemfile and bundle. As long as you have your application setup correctly with Heroku you can then:

git push heroku

On a successful build your output will begin with the following (note the vendoring of http://your-bucket.s3.amazonaws.com/aspell-0.60.6.tgz):

-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
=====> Detected Framework: VendorBinaries
-----> Found a .vendor_urls file
       Vendoring http://your-bucket.s3.amazonaws.com/aspell-0.60.6.tgz
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-ruby.git
=====> Detected Framework: Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using Bundler version 1.3.2
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
       Fetching gem metadata from https://rubygems.org/..........
       Fetching gem metadata from https://rubygems.org/..
       Installing rake (10.1.0)

Test the FFI::Aspell Gem

The moment of truth! Let's run a Rails console to test the FFI:Aspell gem.

heroku run console

At the Rails console:

speller = FFI::Aspell::Speller.new('en_US', 'dict-dir' => '/app/lib/aspell-0.60')
speller.correct?('cookie') # => true

Note we need to specify the dict-dir option since the Aspell library looks for it in /tmp/aspell/lib/aspell-0.60 by default (based on the prefix option we used when we compiled it). If you were to execute the binary directly from the shell you would also need to include this option. For example:

echo "helloz" | aspell -a --dict-dir /app/lib/aspell-0.60

If you don't include the dict-dir option, the FFI:Aspell library will crash.

Troubleshooting

If you have problems, it best to try to run the Aspell library directly from the shell using the example command above.

If you receive the following error:

Error: No word lists can be found for the language "en_US".

Then the path to the dictionary files is not correct.

References

Please note this blog is no longer maintained. Please visit CivilCode Inc - Custom Software Development.