On Building a Debian Package of a Ruby Program

Building a Debian Package from a program written in Ruby is not a straightforward task. This post intends to be a step by step practical guide on packaging ruby programs based on the lessons we learned during the debianization process.

We will use in this guide a sample program: Pagelyzer (http://wiki.opf-labs.org/display/TR/Pagelyzer). This program is an interesting example because of its complexity, it contains Ruby code, java, javascript, as well as some binary libraries in C.

Packaging Ruby scripts is not that different as packaging other software, but using different rules. A debian packaging software relies on standard linux development tools, such as make. This step is crucial to construct a deb package.

As Ruby is normally interpreted (can be compiled but it is not usual) the make command will not work, because there is not MakeFile. Therefore, ruby community has put in place an option for going to the whole process. The proutils ruby project gives all the requirements needed to create a deb package. Its goal is to work in the same way as the make command. Thus, the packaging software won't complain in the process.

This tutorial is a summary based on the previous work of Ubuntu developer David Green tutorial posted on Sep 2012 (https://wiki.ubuntu.com/PackagingGuideDeprecated/Ruby).

Requirements

In this section we will describe the software and file structures needed to make the package.

Setting up the enviroment

Here's what we need to begin packaging our software:

ruby
wget
dh-make
build-essential
fakeroot
cdbs
debhelper
ruby-pkg-tools

The corresponding apt-get command:

$ sudo apt-get install ruby1.9.1-full wget dh-make build-essential fakeroot cdbs debhelper ruby-pkg-tools

Some of the tools introduced below will look for two environment variables to guess your name and email address to put in the package metadata, let's set them up here:

$ export DEBFULLNAME="Your Name"
$ export DEBEMAIL="[email protected]"

you should also add these to your .bashrc or other shell startup script if you want them to be set up automatically.

Creating the Source Archive

To create the source archive we need to:

Create a new directory for the project.
Download setup.rb
Create the directory structure.
Add the files.
Test that it works.
Create the tarball.

Create a Directory

Create a directory in the following format package-name-version. We will use pagelyzer-ruby-0.9

Also, change into the new directory.

$ mkdir pagelyzer-ruby-0.9
$ cd pagelyzer-ruby-0.9

Getting setup.rb

We need to download the setup.rb file from:

$ wget http://i.loveruby.net/archive/setup/setup-3.4.1.tar.gz

We only need the setup.rb file, we can delete the rest of the files in the folder.

Or download it from the attachment (bottom of the page): http://www.openpreservation.org/system/files/setup-3.4.1.zip

Create the Directory Structure

The directory structure used by setup.rb is as followed:

PackageTop/
    lib/
        (ruby scripts)
    ext/
        (ruby extensions)
    bin/
        (commands)
    data/
        (data files)
    etc/
        (configuration files)
    man/
        (manual pages)
    test/
        (tests)

(taken from the setup.rb manual)

Create these directories:

$ mkdir lib ext bin data etc man test

Create other directories that will be used:

$ mkdir man/man1 data/pagelyzer-ruby data/pagelyzer-ruby/js data/doc data/doc/pagelyzer-ruby

Add the Files

Here we list the correspondence of scripts into the directory structure:

File	Folder
pagelyzer_analyzer	bin
pagelyzer_capture	bin
pagelyzer_changedetection	bin
pagelyzer_block.rb	lib
pagelyzer_convex_hull.rb	lib
pagelyzer_dimension.rb	lib
pagelyzer_driver.rb	lib
pagelyzer_heuristic.rb	lib
pagelyzer_point.rb	lib
pagelyzer_separator.rb	lib
pagelyzer_url_utils.rb	lib
pagelyzer_util.rb	lib
js/compress_js.rb	data/pagelyzer-ruby/js
js/decorate.js	data/pagelyzer-ruby/js
js/decorate_mini.js	data/pagelyzer-ruby/js
marcalizer.zip	data/pagelyzer-ruby
pagelyzer_diff.jar	data/pagelyzer-ruby

Note: All .rb files in bin and lib folder should be executable. In contrary case, setup.rb will not include them.

We need to create a manpage for each executable file in /usr/bin. To do this edit man/man1/pagelyzer_changedetection.1. Here a small example, but it should be more extensive.

.TH pagelyzer_changedetection 1 "JAN 20 2013" "Andrés Sanoja"
.SH NAME
pagelyzer_changedetection \- a tool for detecting changes in web pages and their rendering
.SH SYNOPSIS
.B pagelyzer_changedetection
.BR [string]
.PP
.SH DESCRIPTION
Covers the change detection process: capture, segmentation, version analysis (visual and structural)
.PP
.SH AUTHOR
.TP
Andrés SANOJA <[email protected]>
Zeynep Pehlivan <[email protected]>
Myriam Ben Saad <[email protected]>
Marc Law <[email protected]>
Carlos Sureda <[email protected]>
Jordi Creus <[email protected]>

Note: manpages are written in the nroff format. You can also use other formats such as ri or pod and convert them to nroff.

Test That it Works

Install pagelyzer-ruby on your system using setup.rb directly:

$ ruby setup.rb config
$ sudo ruby setup.rb install

Next run:

$ capture.rb –url=http://www.lip6.fr

which should output the web page screenshot, decorated file and source code in the ~/pagelyzer/out folder

Also, test that the manpage works:

$ man pagelyzer_analyzer

To uninstall run:

$ sudo rm -rfi `cat InstalledFiles`

Delete the '.config' file:

$ rm .config

Create the Tarball

Create a gzipped tar archive of the working folder:

$ cd ..
$ tar cavf pagelyzer-ruby-0.9.tar.gz pagelyzer-ruby-0.9

This should create your source archive, pagelyzer-ruby-0.9.tar.gz.

The Packaging Process

To create a package we need to:

Set up the extra files needed for packaging (in the debian/ directory).
Remove the unnecessary files that were created.
Edit the debian/rules file.
Edit the debian/control file.
Edit the debian/postinst file.
Edit the other meta data files.
Build the packages.

Setup the Extra Files

We are going to use dh_make, which will create a template from which we will work on. Run:

$ dh_make -c lgpl -s -r cdbs -f ../pagelyzer-ruby-0.9.tar.gz

which means: -c lgpl tells it that the package is licensed under the LGPL license, -s tells it that we just want one binary package, -r tells it to use CDBS, Common Debian Build System, which will make our packaging simple, so we can concentrate on the Ruby specific things. -f ../pagelyzer-ruby-0.9.tar.gz tells it that we are using the ../pagelyzer-ruby-0.9.tar.gz file as our source.

You should see something like:

Maintainer name : Your Name
Email-Address   : [email protected]
Date            : Wed, 24 Jan 2013 19:53:51 +0530
Package Name    : pagelyzer-ruby
Version         : 0.9
License         : lgpl3
Using dpatch    : no
Using quilt     : no
Type of Package : cdbs
Hit <enter> to confirm:
Currently there is no top level Makefile. This may require additional tuning.

Please edit the files in the debian/ subdirectory now. Before we look at what has happened inside the pagelyzer-ruby-0.9/ directory, let's see what has happened to the directory above it:

$ ls ..

You'll notice that there is a file here that we haven't created: pagelyzer-ruby-1.0.orig.tar.gz. Packaging programs, in addition to binary package, also generates a source package which consists of three files: ${PKGNAME}_${VER}.orig.tar.gz (the original upstream tarball), ${PKGNAME}_${VER}-${PKGVER}.diff.gz (a diff file for the debian/ directory) and ${PKGNAME}_${VER}-${PKGVER}.dsc (a signed summary of the source package). Because we told dh_make where our upstream source tarball was, it renamed it appropriately (${PKGNAME}_${VER}.orig.tar.gz). We could very well have renamed it ourself and not passed the -f option, we chose to be lazy!

Rename debian/postinst.ex file

postinst.ex is a template we need later, rename it to postinst (without extension):

$ mv debian/postinst.ex debian/postinst

Remove Unnecessary Files

Some of the files created are examples and not required. We can delete those with this command:

$ rm debian/*.ex debian/*.EX debian/READ*

Edit debian/rules

Set the contents of debian/rules to this:

#!/usr/bin/make -f
# -*- mode: makefile; coding: utf-8 -*-

include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/ruby-pkg-tools/1/class/ruby-setup-rb.mk

This tells the packager to use setup.rb to create the package.

Edit debian/control

Edit the contents of debian/control to something like this:

Source: pagelyzer-ruby
Section: misc
Priority: extra
Maintainer: Andrés Sanoja <[email protected]>
Build-Depends: cdbs, debhelper (>= 8.0.0), ruby-pkg-tools
# ruby1.9.1-full, libxslt-dev, libxml2-dev, openjdk-7-jdk, imagemagick, ruby1.9.1-dev
Standards-Version: 3.9.2
Homepage: http://wiki.opf-labs.org/display/TR/Pagelyzer
#Vcs-Git: git://git.debian.org/collab-maint/pagelyzer.git
#Vcs-Browser: http://git.debian.org/?p=collab-maint/pagelyzer.git;a=summary

Package: pagelyzer-ruby1.9.1
Architecture: amd64
Depends: ruby1.9.1, cdbs, debhelper (>= 8.0.0), ruby-pkg-tools, libxslt-dev, libxml2-dev, openjdk-6-jdk, imagemagick, ruby1.9.1-dev, ${shlibs:Depends}, ${misc:Depends}
# ruby1.9.1-full
# openjdk-7-jdk
Description: Suite of tools for detecting changes and its rendering
Tool for the web pages comparison based on structural and visual approach.
Research challenge for this tool is the learning algorithm based on frequency.
.
Pagelyzer is a tool which compares two web pages versions and decides if they
are similar or not.
.
It is based on:
* a combination of structural and visual comparison methods embedded in a
statistical discriminative model,
* a visual similarity measure designed for Web pages that improves change
detection,
* a supervised feature selection method adapted to Web archiving.
.
We train a Support Vector Machine model with vectors of similarity scores
between successive versions of pages. The trained model then determines whether
two versions, defined by their vector of similarity scores, are similar or not.
Experiments on real Web archives validate our approach.

Package: pagelyzer-ruby
Architecture: amd64
Depends: pagelyzer-ruby1.9.1, ${misc:Depends}
# , ruby1.9.1-full, cdbs, debhelper (>= 8.0.0), ruby-pkg-tools, libxslt-dev, libxml2-dev, openjdk-6-jdk, imagemagick,ruby1.9.1-dev, ${shlibs:Depends}
# openjdk-7-jdk
Description: Suite of tools for detecting changes and its rendering
metapackage
Suite of tools for detecting changes and its rendering.
Dummy package for pagelyzer-ruby1.9.1

Note that we need to split the packages into a ruby version dependent (dependent on ruby1.9.1) and a dummy package that depends on the version dependent package. If we don't do this, the packaging process will seem to work OK but the packages will not contain any of the files we created will not be in the resulting .deb files! (Remark made by SevenMachines on the Ubuntu Forums thread).

Edit debian/postinst actions

Some ruby gems should be present for the software works properly. In the debian/postinst file (remove .ex extension) add the following:

…
# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.

sudo ln -sf /usr/bin/ruby1.9.1 /usr/bin/ruby
sudo ln -sf /usr/bin/gem1.9.1 /usr/bin/gem

sudo gem install –version '= 0.8.6' hpricot
sudo gem install –version '= 1.5.5' nokogiri
sudo gem install –version '= 2.0.3' sanitize
sudo gem install –version '= 2.29.0' selenium-webdriver

#DEBHELPER#
…

Edit the Other Files

Edit debian/changelog and debian/copyright. Make sure you edit these correctly – especially the debian/copyright file.

Scape project is based on git version control. All changelog information is in there. So, the best way is to download a script from (https://github.com/rackerhacker/gitlog-to-deblog) get into a git hub working folder and generate the changelog file.

It is important to take advice in the version numbers and package name. It should be the same. In our case it is 0.9 but git can change it a bit. For example,

pagelyzer (initial-11-gbbcc12f) unstable; urgency=low

* Including performance test and enhacements in change_detection.rb

should be change to something like this:

pagelyzer-ruby (0.9-11-gbbcc12f) unstable; urgency=low

* Including performance test and enhacements in change_detection.rb

And an example of 'copyright' file:

Format: http://dep.debian.net/deps/dep5
Upstream-Name: pagelyzer-ruby
Source: https://github.com/openplanets/pagelyzer

Files: *
Copyright: 2011, 2012 Andrés Sanoja <[email protected]>
           2011, 2012 Stéphane Gançarski <[email protected]>
           2011, 2012 Zeynep Pehlivan <[email protected]>
           2011, 2012 Denis Pitzalis <[email protected]>
           2011, 2012 Marc Law <[email protected]>
License: LGPL-3.0+

Files: debian/*
Copyright: 2013 Jordi Creus Tomàs <[email protected]>
License: LGPL-3.0+

License: LGPL-3.0+
This package is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 3 of the License, or (at your option) any later version.
.
This package is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
.
On Debian systems, the complete text of the GNU Lesser General
Public License can be found in "/usr/share/common-licenses/LGPL-3".

Build the Packages

Make sure you are in the project root directory and use the 'debuild' command to create the packages:

$ debuild -us -uc

That will build the source and binary package. The -us -uc options are not to sign the source and changes files (we would need to sign them to upload them to Ubuntu/Debian/your PPA, but we'll skip that in this tutorial, if you are interested in PPAs, there was a session about it in https://wiki.ubuntu.com/MeetingLogs/openweekhardy/LaunchpadPPAs

It should create separate packages for pagelyzer-ruby1.9.1_0.9-rr-cccc_i386.deb and pagelyzer-ruby_0.9-rr-cccc_i386.deb. Where rr is the last revision number (from changelog file) and cccc the hash of the revision (also from changelog file)

For some strange reason this command is not enough, we should use also:

$ dpkg-buildpackage

Note: be careful if you use a virtual machine (e.g., virtualbox, etc), your files should be in a folder where you have “real” write permissions (e.g., a folder inside your virtual disk). Otherwise, if you use a shared folder for your files you will come across a 'Read-only file system' error.

Signing your package

You need first a GPG key. Follow the steps in http://keyring.debian.org/creating-key.html. If you have never run gpg before, do it:

$ gpg

This will create the ~/.gnupg directory, then you will be able to modify the ~/.gnupg/gpg.conf file according to the tutorial. Finally, do not forget to make your new key publicly available to the pgp server (it will be automatically distributed on the pgp network in a few minutes):

$ gpg –keyserver subkeys.pgp.net –send-key 12345678

Now, you can finally sign your package by running the command:

$ debuild -k12345678

Copying GPG keys across different machines

If you are building a package for different architectures, i386, amd64… You must sign all them with the same key. To copy a GPG key from one machine to another, you simply need to copy all *.gpg files (pubring.gpg, secring.gpg and trustdb.gpg) and gpg.conf on ~/.gnupg directory from one machine to the other one. (random_seed file is not mandatory)

Uploading your package to the repository

Install and configure dupload according to http://wiki.opf-labs.org/display/SP/Submitting+Your+Package tutorial. Do not forget to enable FTP_PASSIVE mode!

$ export FTP_PASSIVE=1

Finally, you can upload your files:

$ dupload pagelyzer-ruby_0.9-12-gbbcc12f_amd64.changes
…
$ dupload pagelyzer-ruby_0.9-12-gbbcc12f_i386.changes
…

Installing the software

Double click in the .deb file

pagelyzer-ruby1.9.1_0.9-xx-yyyyyyy_i386.deb

This will install everything you need as well as the tool itself.

Authors: Andrés Sanoja & Jordi Creus