Recently I needed to install the Aspell library on Heroku for a side project. By default a Heroku Dyno doesn't include any non-essential libraries so your left to install these on your own. This introduced me to the world of buildpacks and an awesome build tool that Heroku provides to get the job done.
Objective
The objective is to have a Rails application running with FFI::Aspell, a FFI binding for the Aspell library on Heroku.
Heroku Buildpacks and Vendoring Binaries
First, we need to configure Heroku to vendor binaries. The easiest way to do this is use the Vendor Binaries buildpack which allows you to extract a tarball stored on S3 into your /app directory when the application build process is triggered. This is all configured in a .vendor_urls located in the root of your application.
As Heroku only supports a single buildpack by default, we need to configure it to support multiple buildpacks. In our case, we need the Ruby buildpack in addition to the Vendor Binaries buildpack. Fortunately this is relatively easy to do when we create our new application (and can be done after the fact as well):
heroku create --stack cedar --buildpack https://github.com/dollar/heroku-buildpack-multi.git
Now we have our multi buildpack application, we can configure those buildpacks in .buildpacks:
https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git https://github.com/heroku/heroku-buildpack-ruby.git
For our vendored binaries, although we haven't built it yet, we can configure it's location on S3 in .vendor_urls:
http://your-bucket.s3.amazonaws.com/aspell-0.60.6.1.tar.gz
Commit both the .buildpacks and the .vendor_urls files to your Rails application.
Build the binary
Next we need to build the binary. First, let's download and extract the source into a temporary working directory (not in your Rails application).
mkdir ~/Code/temp && cd ~/Code/temp curl -o aspell-0.60.6.1.tar.gz ftp://ftp.gnu.org/gnu/aspell/aspell-0.60.6.1.tar.gz tar -xvzf aspell-0.60.6.1.tar.gz
Now, let's create a build server on Heroku. The Vulcan gem automates the process of creating the server and building the library.
gem install vulcan vulcan create vulcan-yourname
Now we're ready to build the binary:
vulcan build -s ~/Code/temp/aspell-0.60.6.1 -p /tmp/aspell -c "./configure --prefix=/tmp/aspell && make install"
You will notice that we configure the library to be installed in /tmp/aspell using the prefix option. We also need to tell Vulcan using the -p option where the compiled library will be located. At the completion of the build, a tarball is then download to your /tmp directory.
You can now copy the tarball from /tmp/aspell-0.60.6.tgz to the S3 bucket specified in .vendor_urls. Ensure the read permission on the uploaded file can viewed by "world", otherwise it won't be accessible when you deploy your application and you will receive the following errors:
-----> Found a .vendor_urls file
Vendoring http://yourbucket-heroku.s3.amazonaws.com/aspell-0.60.6.tgz
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
! Push rejected, failed to compile Multipack app
Build the supporting dictionary files
Typically we would be done at this point, but we need to compile the dictionary files that Aspell uses separately. This makes the process a little more complicated.
We're going to use our Vulcan build server to compile the dictionary files. This requires starting up a shell, downloading the Aspell tarbar we built previously and extracting it into the original installation directory.
heroku run bash --app vulcan-yourname cd /tmp mkdir aspell && cd aspell curl -o aspell-0.60.6.tgz http://yourbucket-heroku.s3.amazonaws.com/aspell-0.60.6.tgz tar -xzvf aspell-0.60.6.tgz
Now we're ready to build the dictionary files:
export PATH=$PATH:/tmp/aspell/bin curl -o aspell6-en-7.1-0.tar.bz2 ftp://ftp.gnu.org/gnu/aspell/dict/en/aspell6-en-7.1-0.tar.bz2 tar -xvf aspell6-en-7.1-0.tar.bz2 cd aspell6-en-7.1-0 ./configure && make install
At this point you will see output similar to this:
/tmp/aspell/bin/prezip-bin -d < en-common.cwl | /tmp/aspell/bin/aspell --lang=en create master ./en-common.rws /tmp/aspell/bin/prezip-bin -d < en-variant_0.cwl | /tmp/aspell/bin/aspell --lang=en create master ./en-variant_0.rws /tmp/aspell/bin/prezip-bin -d < en-variant_1.cwl | /tmp/aspell/bin/aspell --lang=en create master ./en-variant_1.rws /tmp/aspell/bin/prezip-bin -d < en-variant_2.cwl | /tmp/aspell/bin/aspell --lang=en create master ./en-variant_2.rws ...
Once the build is completed you can test Aspell is working correctly with:
echo "helloz" | aspell -a
We're almost there. Now just copy the dictionary files over to our original build for Aspell in /lib/aspell-0.60/ We will create a new tarball which includes the dictionary files:
cp *.rws *.alias *.multi *.dat ../lib/aspell-0.60/. cd .. tar -czvf aspell-0.60.6.tgz bin include lib share
Finally, transfer the new tarball aspell-0.60.6.tgz to your local machine and upload to S3 again replacing our original tarball.
Deployment
Before deploying you will need to configure LD_LIBRARY_PATH, the path used by the dynamic loader to load libraries into dynamically linked executables. Otherwise, the following error is raised:
aspell: error while loading shared libraries: libaspell.so.15: cannot open shared object file: No such file or directory
This is easy to do:
heroku config:add LD_LIBRARY_PATH=/app/lib
Now let's deploy our application. Remember to include the FFI::Aspell in our Gemfile and bundle. As long as you have your application setup correctly with Heroku you can then:
git push heroku
On a successful build your output will begin with the following (note the vendoring of http://your-bucket.s3.amazonaws.com/aspell-0.60.6.tgz):
-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
=====> Detected Framework: VendorBinaries
-----> Found a .vendor_urls file
Vendoring http://your-bucket.s3.amazonaws.com/aspell-0.60.6.tgz
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-ruby.git
=====> Detected Framework: Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using Bundler version 1.3.2
Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
Fetching gem metadata from https://rubygems.org/..........
Fetching gem metadata from https://rubygems.org/..
Installing rake (10.1.0)
Test the FFI::Aspell Gem
The moment of truth! Let's run a Rails console to test the FFI:Aspell gem.
heroku run console
At the Rails console:
speller = FFI::Aspell::Speller.new('en_US', 'dict-dir' => '/app/lib/aspell-0.60')
speller.correct?('cookie') # => true
Note we need to specify the dict-dir option since the Aspell library looks for it in /tmp/aspell/lib/aspell-0.60 by default (based on the prefix option we used when we compiled it). If you were to execute the binary directly from the shell you would also need to include this option. For example:
echo "helloz" | aspell -a --dict-dir /app/lib/aspell-0.60
If you don't include the dict-dir option, the FFI:Aspell library will crash.
Troubleshooting
If you have problems, it best to try to run the Aspell library directly from the shell using the example command above.
If you receive the following error:
Error: No word lists can be found for the language "en_US".
Then the path to the dictionary files is not correct.
