29 Aug 2013, 12:37

Deploying Perl Web-Apps with Puppet

Since I was asked for I’d like to show one way of how you can deploy your perl web apps using debian packages and puppet.

This post assumes that you want to install some Plack webapps, e.g. App::Standby, Monitoring::Spooler and Zabbix::Reporter.

Those are pretty straight-forward Plack Apps built using Dist::Zilla.

cd Monitoring-Spooler
dzil build
I’ve built debian packages of these CPAN modules. Using dh-make-perl this is reasonably easy, although you’ll need some switches to make dh-make-perl a little less picky:
dh-make-perl –vcs rcs –source-format “3.0 (native)” Monitoring-Spooler-0.04
After building and uploading these packages to my repository I continue to use Puppet to install and configure those.

For each of these packages there is a custom puppet module, e.g. app_standby for managing App::Standby. This module create all necessary directories, manages the configuration and installs the package from my repository. It does not, however, set up any kind of webserver or runtime wrapper. This is done by another module: starman.

For running my Plack apps I use Starman which is managed by it’s own puppet module. It will install the starman package, manage it’s config and create an apache config. The interesting part is that I use puppet to automatically create an apache config and a simple Plack/PSGI multiplexer app that allows me to handle requests to all three (or more) of these apps using a single starman instance.

If these apps were rather busy I’d use a separate starman instance for each one, but as it is I’m more than happy to avoid the overhead of running multiple starman instances.

My starman module is invoked from one of my service classes like this:

  class { ‘starman’:
    configure => {
      mounts    => {
        ‘zreporter’     => ‘/usr/bin/zreporter-web.psgi’,
        ‘monspooler’    => ‘/usr/bin/mon-spooler.psgi’,
        ‘monapi’        => ‘/usr/bin/mon-spooler-api.psgi’,
        ‘standby’       => ‘/usr/bin/standby-mgm.psgi’,
      },
    }
  }
The starman module itself is pretty straight-forward, but below are the templates for the apache config and the PSGI multiplexer.

apache.conf.erb:

<IfModule mod_proxy.c>
<% if @configure.has_key?(“mounts”) -%>
<% @configure[“mounts”].keys.sort.each do |mount| -%>
   <Location /<%= mount %>>
      Allow from all
      ProxyPass           http://localhost:<%= @configure.has_key?(‘starman_port’) ? @configure[“starman_port”] : ‘5001’ %>/<%= mount %>
      ProxyPassReverse    http://localhost:<%= @configure.has_key?(‘starman_port’) ? @configure[“starman_port”] : ‘5001’ %>/<%= mount %>
   </Location>
<% end -%>
<% end -%>
</IfModule>
mount.psgi.erb:
#!/usr/bin/perl
use strict;
use warnings;

use Plack::Util; use Plack::Builder;

my $app = builder { <% if @configure.has_key?(“mounts”) -%> <% @configure[“mounts”].keys.sort.each do |mount| -%>    mount ‘/<%= mount %>‘  => Plack::Util::load_psgi(’<%= @configure[“mounts”][mount] %>’); <% end -%> <% end -%>    mount ‘/‘           => builder {         enable ‘Plack::Middleware::Static’, path => qr#/.+#, root => ‘/var/www’;         my $app = sub { return [ 302, [‘Location’,‘/index.html’], [] ]; };         $app;    } };

 

02 Oct 2011, 16:13

VBoxAdm 0.1.15

I’ve just uploaded VBoxAdm 0.1.15. It includes another set of major refactorings. Please be careful when upgrading and watch your logfiles closely for any errors that may occur. Especially the Vacation module was refactored.

The time when the project will leave it’s alpha stage is drawing closer. VBoxAdm is now running on several largish sites under my direct administrative control, so I’ve got plenty of possiblities for some real-world testing. Several other migrations/installations are planned for the near future. Once it has proven sufficiently stable on these mailservers I’ll announce the the end of the alpha phase and enter beta testing.

Stay tuned!

16 Apr 2011, 13:14

VBoxAdm: Mailinglist and API

A short status update regarding VBoxAdm.

Mailinglist

Finally I’ve created a Mailinglist: http://www.vboxadm.net/support.html#mailinglist

Refactoring

I’ve been refactoring the Code for a while to turn it more into a MVC-Shape. This means separating the Model from the Controller (former VBoxAdm::Frontend, now VBoxAdm::Controller::Frontend). The ultimate goal of this work is to support code reuse and support for multiple ways to manipulate the data. Once the Model classes are stable I’ll finish the command line interface as well as the HTTP-API. This will provide three ways to modify the underlying data:
  • Web Frontend
  • HTTP-API (no REST for now, maybe later)
  • CLI
The Mailarchive is postponed for the time being.

Auto Configuration

Most Mailclients, like Outlook, Thunderbird and KMail, support a way of client auto-configuration. When setting up a new mail account they request a certain URL derived from the mail address and if they find an XML document with the expected information there they’ll use this information to set the correct username, mailserver and protocols. Support for this was added recently. There is even support for the weird way MS Outlook does this. However Outlook support is, so far, based solely on the documentation on Technet. Due to the lack of a Outlook license I wasn’t able to test it. Please provide feedback.

Future Work

After the refactoring, API and CLI are finished I’m going to look into the Mailarchive again. After that I’ll look into Quota Support, Mailman integration and I’d like to find a way to get the Postfix Logs into the database to ease support work. Having the Log in the database in a parsed format - no raw syslog to db - would make support request more easy to handle. No more need to log into the server and grep through the mail.log.

Further feature request are always welcome. Please direct any ideas and comments to the mailinglist at vboxadm@vboxadm.net.

19 Dec 2010, 17:38

Handling salted passwords in Perl

While working on VBoxAdm I’ve came to a point where I did want to support salted passwords. Salted passwords however are a bit difficult since you need to extract the salt from the existing password hash and compare it to the user supplied password to hash it and compare the result. You can’t simply hash the supplied pass and use a random salt, that would produce different hashes.

If you’re not into salted hashes, I’ll try to explain how salted hashes are generated, why they are and how you can handle them.

What are salted hashes and why do you need them?

The simplest way to store passwords in a database would be to use plaintext. Let’s assume for the remainder of this post that we’ll use the password ‘password’ and the salt ‘salt’. If you use plaintext you just store the string ‘password’ in your database. That is easy to implement since you don’t need to do anything with the password and this gives you much flexibility since you can do anything with the password, e.g. support challenge response authentication schemes. However this has a major drawback: The (database) administrator and perhaps others, as well as any intruders that gain access to your database can see all users passwords. In a perfect world this wouldn’t be a big problem since all users should use a unique password for every service they use. Yet in the real world there is small but important problem: Many users will re-use their passwords for other services. This would allow an intruder to abuse the passwords to access (maybe) much more important accounts of your users and cause much trouble to them and possibly bad PR to you.

It’d be much better if the passwords would be encrypted in some way so the cleartext is not directly visible from the database. This would help with security and give your users more privacy. At least I would not want that anybody knows what passwords I use. Not even the administrator of the site I use the password for. This is were hashing comes into play. Hashes are cryptographic one-way functions that have the ability to produce an (almost) unique output for a given input. The same input always produces and same output and it is a very rare occasion that two different inputs produce the same output (which would be called a collision).  This solves much of the initial problem, but still some issues remains: If two users use the same password this would be immediately visible because they would have the same hashes. Another issue is that you can easily lookup the cleartext for a given hash in a rainbow table. I’m sure that at least intelligence services will have huge rainbow tables.

Luckily someone came up with the idea of salted hashes. A salted hash is generated by creating a random salt for every hashing operation and appending this salt onto the cleartext. Since there must be a way to compare hashes later the salt is usually appended to the resulting hash and can be extracted from there for subsequent hashing and comparison. This makes salted hashes pretty much safe against a wide area of attacks and meets many privacy requirements.

How to deal with these salted hashes in perl?

I’ll use the salted hashes created by Dovecots dovecotpw utility as an example since these were the hashes of my concern and they aren’t to most simple ones. These hashes are created by generating a random salt, appending it to the cleartext and appending the salt to the hash.
SSHA(password, salt) = SHA(password, salt) + salt
The SHA1 hash uses 20 bytes of (binary) data for the hash. The salt is variable length and consists of binary data. This is important: The salt is not just ASCII (7-bit) or UTF-8 (variable lenght, limited value range) but uses the full eight bit of each byte to allow for maximum entropy. This is important. If you doubt it get a good book on cryptograhpy and read up on the issue of randomness, entropy and cryptanalysis. The problem here is that perl is tailored to deal with text data and numbers but not binary data. I C this would be an easy task, but perl needs special care to handle this requirements.

Generating an salted hash is pretty easy: Generate a number of valid byte values (0-254) and pack() it into a binary string, e.g.:

sub make_salt {
    my $len   = 8 + int( rand(8) );
    my @bytes = ();
    for my $i ( 1 .. $len ) {
        push( @bytes, rand(255) );
    }
    return pack( ‘C*‘, @bytes );
}
However if you want to compare a user-supplied pass with the salted hash in your database you have to take some more steps. You’ll first need to retreive the old salted hash from the database, extract the salt and hash the user supplied password toegether with this salt. If you’d use another salt the resulting hashes would be different and the user would not be able to login although the passwords match.

In my implementation I did separate the code into a split_pass method that dissects the old hash and extracts the salt (and possibly the password scheme used) and a make_pass method that takes a cleartext, a password scheme and a salt to generate a hashed password. The resulting hash is them compared with the stored hash and if they match the user may login or do whatever the authorization was requested for.

The split_pass method basically strips of the password scheme stored in front of the hash between curly brackets, decodes the Base64 encoded hash, unpack()s it and packs everything after the hash again into a binary string using pack(‘C*‘,@salt). The pack() template represents a one byte char, which is what we need here.

For the actual implementation I suggest you look at the Perl Module VBoxAdm::DovecotPW distributed with VBoxAdm.

package VBoxAdm::DovecotPW;

use strict;
use warnings;

use MIME::Base64;
use Digest::MD5;
use Digest::SHA;

our $VERSION = '@VERSION@';

my %hashlen = (
    'smd5'    => 16,
    'ssha'    => 20,
    'ssha256' => 32,
    'ssha512' => 64,
);

############################################
# Usage      : my $hash = VBoxAdm::DovecotPW::plain_md5('pwclear');
# Purpose    : ????
# Returns    : ????
# Parameters : ????
# Throws     : no exceptions
# Comments   : none
# See Also   : http://wiki.dovecot.org/Authentication/PasswordSchemes
sub plain_md5 {
    my $pw = shift;
    return "{PLAIN-MD5}" . Digest::MD5::md5_hex($pw);
}

sub ldap_md5 {
    my $pw = shift;
    return "{LDAP-MD5}" . pad_base64( Digest::MD5::md5_base64($pw) );
}

sub smd5 {
    my $pw = shift;
    my $salt = shift || &make_salt();
    return "{SMD5}" . pad_base64( MIME::Base64::encode( Digest::MD5::md5( $pw . $salt ) . $salt, '' ) );
}

sub sha {
    my $pw = shift;
    return "{SHA}" . MIME::Base64::encode( Digest::SHA::sha1($pw), '' );
}

sub ssha {
    my $pw = shift;
    my $salt = shift || &make_salt();
    return "{SSHA}" . MIME::Base64::encode( Digest::SHA::sha1( $pw . $salt ) . $salt, '' );
}

sub sha256 {
    my $pw = shift;
    return "{SHA256}" . MIME::Base64::encode( Digest::SHA::sha256($pw), '' );
}

sub ssha256 {
    my $pw = shift;
    my $salt = shift || &make_salt();
    return "{SSHA256}" . MIME::Base64::encode( Digest::SHA::sha256( $pw . $salt ) . $salt, '' );
}

sub sha512 {
    my $pw = shift;
    return "{SHA512}" . MIME::Base64::encode( Digest::SHA::sha512($pw), '' );
}

sub ssha512 {
    my $pw = shift;
    my $salt = shift || &make_salt();
    return "{SSHA512}" . MIME::Base64::encode( Digest::SHA::sha512( $pw . $salt ) . $salt, '' );
}

sub make_pass {
    my $pw     = shift;
    my $scheme = shift;
    my $salt   = shift || &make_salt();
    if ( $scheme eq 'ldap_md5' ) {
        return &ldap_md5($pw);
    }
    elsif ( $scheme eq 'plain_md5' ) {
        return &plain_md5($pw);
    }
    elsif ( $scheme eq 'sha' ) {
        return &sha($pw);
    }
    elsif ( $scheme eq 'sha256' ) {
        return &sha256($pw);
    }
    elsif ( $scheme eq 'sha512' ) {
        return &sha512($pw);
    }
    elsif ( $scheme eq 'smd5' ) {
        return &smd5( $pw, $salt );
    }
    elsif ( $scheme eq 'ssha' ) {
        return &ssha( $pw, $salt );
    }
    elsif ( $scheme eq 'ssha256' ) {
        return &ssha256( $pw, $salt );
    }
    elsif ( $scheme eq 'ssha512' ) {
        return &ssha512( $pw, $salt );
    }
    else {
        return "{CLEARTEXT}" . $pw;
    }
}

sub make_salt {
    my $len   = 8 + int( rand(8) );
    my @bytes = ();
    for my $i ( 1 .. $len ) {
        push( @bytes, rand(255) );
    }
    return pack( 'C*', @bytes );
}
# this method was copied from some module on CPAN, I just don't remember which one right now
sub pad_base64 {
    my $b64_digest = shift;
    while ( length($b64_digest) % 4 ) {
        $b64_digest .= '=';
    }
    return $b64_digest;
}

sub verify_pass {

    # cleartext password
    my $pass = shift;

    # hashed pw from db
    my $pwentry = shift;

    my ( $pwscheme, undef, $salt ) = &split_pass($pwentry);

    my $passh = &make_pass( $pass, $pwscheme, $salt );

    if ( $pwentry eq $passh ) {
        return 1;
    }
    else {
        return;
    }
}

sub split_pass {
    my $pw       = shift;
    my $pwscheme = 'cleartext';

    # get use password scheme and remove leading block
    if ( $pw =~ s/^\{([^}]+)\}// ) {
        $pwscheme = lc($1);

        # turn - into _ so we can feed pwscheme to make_pass
        $pwscheme =~ s/-/_/g;
    }

    # We have 3 major cases:
    # 1 - cleartext pw, return pw and empty salt
    # 2 - hashed pw, no salt
    # 3 - hashed pw with salt
    if ( !$pwscheme || $pwscheme eq 'cleartext' || $pwscheme eq 'plain' ) {
        return ( 'cleartext', $pw, '' );
    }
    elsif ( $pwscheme =~ m/^(plain-md5|ldap-md5|md5|sha|sha256|sha512)$/i ) {
        $pw = MIME::Base64::decode($pw);
        return ( $pwscheme, $pw, '' );
    }
    elsif ( $pwscheme =~ m/^(smd5|ssha|ssha256|ssha512)/ ) {

        # now get hashed pass and salt
        # hashlen can be computed by doing
        # $hashlen = length(Digest::*::digest('string'));
        my $hashlen = $hashlen{$pwscheme};

        # pwscheme could also specify an encoding
        # like hex or base64, but right now we assume its b64
        $pw = MIME::Base64::decode($pw);

        # unpack byte-by-byte, the hash uses the full eight bit of each byte,
        # the salt may do so, too.
        my @tmp  = unpack( 'C*', $pw );
        my $i    = 0;
        my @hash = ();

        # the salted hash has the form: $saltedhash.$salt,
        # so the first bytes (# $hashlen) are the hash, the rest
        # is the variable length salt
        while ( $i < $hashlen ) {
            push( @hash, shift(@tmp) );
            $i++;
        }

        # as I've said: the rest is the salt
        my @salt = ();
        foreach my $ele (@tmp) {
            push( @salt, $ele );
            $i++;
        }

        # pack it again, byte-by-byte
        my $pw   = pack( 'C' . $hashlen, @hash );
        my $salt = pack( 'C*',           @salt );

        return ( $pwscheme, $pw, $salt );
    }
    else {

        # unknown pw scheme
        return;
    }
}

1;