Mercurial escaped colors

After upgrading Mercurial to 4.2 on my FreeBSD 10.x boxes, there was a problem in that the mercurial color extension was now enabled, and suddenly things like hg status were showing output like

ESC[0;34;1mM ESC[0mESC[0;34;1mpf.confESC[0m

after lots of digging, finally figured out it was caused by my PAGER environment variable being set to more, pretty outdated. Fixed it on-the-fly with export PAGER='less -X' and got nice colorized output. Made it permanent by editing ~/.profile and replacing a line I had setting the PAGER with a new one:

PAGER='less -X';    export PAGER

Winbind failure do to incorrect time

I had the weirdest thing suddenly start happening last night that took several hours to finally figure out was a time-related issue.

I've got an Ubuntu box that uses pam_winbind to allow for logging into a machine using an Active Directory account.
Normally I connect with an SSH key, but once in when doing sudo -s I enter an AD password to become root. Last night that sudo -s suddenly stopped working.

Luckily I had another non-AD account that I could connect with, and sudo worked for that, so I could become root and poke around. The logs showed:

sudo: pam_unix(sudo:auth): authentication failure; logname=barry.pederson uid=14283 euid=0 tty=/dev/pts/0 ruser=barry.pederson rhost=  user=barry.pederson
sudo: pam_unix(sudo:auth): conversation failed
sudo: pam_unix(sudo:auth): auth could not identify password for [barry.pederson]

That was weird, I could log into other things though that used the same AD account, so I knew the password was right and the account wasn't locked out.

I hoped by the next morning, some cache thing would expire and I'd be back in business, but no dice.

Poking around some more I found if I disabled my SSH keys, I couldn't log in at all, so it was really a pam_winbind issue, not a sudo one. The logs for a SSH password login attempt were a bit more informative:

pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.yyy.zzz  user=barry.pederson
pam_winbind(sshd:auth): getting password (0x00000388)
pam_winbind(sshd:auth): pam_get_item returned a password
pam_winbind(sshd:auth): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_AUTH_ERR (7), NTSTATUS: NT_STATUS_LOGON_FAILURE, Error message was: Logon failure
pam_winbind(sshd:auth): user 'barry.pederson' denied access (incorrect password or invalid membership)
Failed password for barry.pederson from x.x.x.x port 50655 ssh2

WTF? I know the password's right, I've been typing it all morning into other systems. I even tried wbinfo --authenticate barry.pederson on this box and it accepted my passwords.

Much time was spent Googling, trying various tweaks to smb.conf, etc. Finally, I don't remember why, I thought to check the date with ntpdate -d my.ad.server and it came back with offset -338.308573 sec. Holy crap, that's more than 5 minutes! Even though ntpd is running.

Anyhow, once the clock was fixed to be closer to the AD server, logins and sudo started working again.

HTTPS Now

Put in a actual, recognized SSL Certificate on the site, and setup redirects to run everything through that now.

Figured that was a reasonable thing to do because people are still occasionally downloading old software from this site, and the cert was free for the year (Gandi).

Hopefully by the time it expires the Let's Encrypt service will be up and running.

Named routes in Laravel 4

I've been doing some work with PHP & Laravel 4, and had an idea to make naming routes a bit nicer syntactically. Currently, to name a route you wrap the 2nd parameter in an array and add an item with an 'as' key, as in:

Route::get('user/profile', array('as' => 'profile', function(){// code here..}));

I think it'd be more natural to set the name using a method on the route (similar to adding a where() condition) , something like:

Route::get('user/profile', function(){// code here..})
    ->named('profile');

After a bit of poking around, I found this could be done fairly easily with a couple small additions to Route and Router to allow for renaming a route:

--- a/vendor/laravel/framework/src/Illuminate/Routing/Route.php Fri Jan 25 15:35:04 2013 -0600
+++ b/vendor/laravel/framework/src/Illuminate/Routing/Route.php Sat Jan 26 16:37:08 2013 -0600
@@ -411,4 +411,16 @@
                return $this;
        }

-}
+    /**
+     * Change the name for this route.
+     *
+     * @param string $name New Name
+     * @return Illuminate\Routing\Route
+     */
+    public function named($name)
+    {
+        $this->router->rename($this, $name);
+
+        return $this;
+    }
+}
\ No newline at end of file
diff -r 50adf81e2f0f vendor/laravel/framework/src/Illuminate/Routing/Router.php
--- a/vendor/laravel/framework/src/Illuminate/Routing/Router.php        Fri Jan 25 15:35:04 2013 -0600
+++ b/vendor/laravel/framework/src/Illuminate/Routing/Router.php        Sat Jan 26 16:37:08 2013 -0600
@@ -1159,4 +1159,21 @@
                $this->container = $container;
        }

+    /**
+     * Change the name of a route
+     *
+     * @param $route
+     * @param $name
+     * @return void
+     */
+    public function rename($route, $name)
+    {
+        foreach($this->routes->getIterator() as $n => $r) {
+            if ($r === $route) {
+                $this->routes->remove($n);
+                $this->routes->add($name, $r);
+                return;
+            }
+        }
+    }
 }

It's not the most efficient thing in the world to be iterating over all the previous routes. A small optimization would be to check the last-added route first, in the example usage that'd be the one you're renaming. But to do that would require also patching Symfony's RouteCollection class.

Integrated Windows Authentication with Apache on a Linux box

Integrated Windows Authentication (IWA) is a useful feature for intranets, where a web browser on a Windows client joined to Active Directory (AD) can seamlessly pass authentication information to a web server - without needing to prompt the user for a password. It's supported by IE, Firefox and Chrome on the client side, and naturally by IIS on the server side. With just a little bit of effort, it can also be supported by Apache on a Linux or other Unix-type OS, and I'll take a look at doing that here.

IWA is a generic term that covers a few different protocols. One is the older NTLM Authentication, which can be setup on a Linux server with mod_auth_ntlm_winbind, but that's awfully clunky and requires setting up and running Samba's winbindd daemon. Another is Kerberos, which is fairly well supported in must Linux/Unix distros, and is an integral part of Active Directory.

There are a lot of writeups on integrating a Linux box with AD, but most of them get very complicated, trying to integrate everything, including login, file sharing, group mapping, etc. And deeply relying on Samba. I'm going to focus on just the one simple task of HTTP authentication in Apache, not using Samba, and being as explicit as possible on what needs to be done on both the Linux and Windows Active Directory sides of the setup.

Prerequisites

I'm going to do this for Ubuntu 10.04 and assume you have root access and are familiar with general Apache configuration. Other Linux distros or perhaps BSDs should be very very similar.

Some other things you're going to need to be able to do, or at least get someone in you organization to do for you are:

  • Create an AD User object
  • Run the setspn and ktpass Windows commandline utilities against the User object you create
  • Setup forward and corresponding reverse DNS records for your server.

Examples

For the rest of this, we're going to assume:

  • AD Domain name = ad.foobar.edu
  • AD Kerberos Realm = AD.FOOBAR.EDU (usually the domain-name uppercased)
  • Our Linux box's domain name = test.foobar.edu
  • Linux box's IP = 1.2.3.4
  • Reverse lookup of 1.2.3.4 results in test.foobar.edu

AD Setup

Firstly, we need a User object in Active Directory that will represent the Apache service, and will hold a password which Kerberos tickets will be based on.

In the Active Directory Users and Computers utility, create a User object, the name doesn't matter much, so I'll go with Test-HTTP

User object creation

after hitting Next >, on the password page uncheck User must change password... and check Password never expires. Go ahead and enter anything as a password, it'll get changed to something random in a later step.

User object password

Go ahead and finish that up.

Next, we need to associate a Service Principal Name (SPN) with the User object we just created. Kerberos principals are usually <protocol>/<domain-name>@<kerberos-realm>. Since we're doing a web server, it'll be known in Kerberos as HTTP/test.foobar.edu@AD.FOOBAR.EDU Run this in a Command Prompt window:

setspn -A HTTP/test.foobar.edu Test-HTTP

(note that we left off the @AD.FOOBAR.EDU part, setspn knows to put that in)

Lastly, we're going to create a keytab file (I'll call it test-http.keytab), which holds encryption keys based on the User object's password. When a client requests a ticket to access our Linux box, AD will locate the User object based on the SPN we associated with it, and use the same encryption keys to create the Kerberos tickets our Linux's Apache will be setup to require.

(This is a one-line command, but I'm going to display it below as several lines for readability)

ktpass -out test-http.keytab 
    -princ HTTP/test.foobar.edu@AD.FOOBAR.EDU 
    -mapuser Test-HTTP 
    -mapOp set 
    +rndPass 
    -crypto All 
    -ptype KRB5_NT_PRINCIPAL

The +rndPass changes the User objects password to something random, you don't need to know what it is - the keytab is the thing you really care about here.

Securely copy that test-http.keytab to the Linux box, and delete it off the Windows machine. We're done with AD now, back to the real world...

Linux setup

Move the keytab file somewhere handy, such as /etc/apache2/test-http.keytab, and set the permissions so that the Apache process (and nobody else) has access:

chmod 440 test-http.keytab
chown www-data:www-data test-http.keytab

Install the Apache Kerberos module

aptitude install libapache2-mod-auth-kerb

You'll need an /etc/krb5.conf file. A simple one that leaves it up to Kerberos to discover what it needs might be as simple as:

[libdefaults]
default_realm = AD.FOOBAR.EDU

Here's a more explicit one that specifies Active Directory KDCs (Key Distrubution Centers), by IP

[libdefaults]
default_realm = AD.FOOBAR.EDU
default_keytab_name = FILE:/etc/krb5.keytab

[realms]
AD.FOOBAR.EDU = {
    kdc = 1.2.0.1
    kdc = 1.2.0.2
    kdc = 1.2.0.3
    default_domain = AD.FOOBAR.EDU
    }

[domain_realm]
.foobar.edu = AD.FOOBAR.EDU

That sort of thing is documented on the MIT website.

Apache Setup

We're in the home stretch now, Apache directives to protect a cgi-bin directory for example might look like:

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
    AllowOverride None
    Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
    Order allow,deny
    Allow from all

    AuthName "FOOBAR Active Directory"
    AuthType KerberosV5
    KrbServiceName HTTP
    Krb5Keytab /etc/apache2/test-http.keytab
    require valid-user
</Directory>

Those last 5 lines inside the <Directory> block are the key here. The KrbServiceName of HTTP corresponds to what we entered as the protocol part of the principal name back on the setspn and ktpass commands. The AuthName is what will be displayed if the browser falls back to basic autentication if Kerberos is not available.

Test it out

Here's a super-simple CGI test.sh script we can put in our Kerberos-protected cgi-bin directory, be sure to make it executable.

#!/bin/sh
echo 'Content-Type: text/plain' 
echo
echo You are: $REMOTE_USER

Go to a Windows client signed into Active Directory. To get IE or Chrome to attempt Kerberos authentication you'll have to add test.foobar.edu to the Local Intranet in the Internet Settings control panel. Here are some shots of where to go:

Local Intranet screenshot 1

Local Intranet screenshot 1

Local Intranet screenshot 1

For Firefox, you'll want the NTLMAuth add-in, which lets you specify which domains that Firefox should attempt Kerberos authentication with.

Once you've got the browser fixed up, try accessing http://test.foobar.edu/cgi-bin/test.sh, and if everything works out, you should be rewarded with something like:

You are: bob.smith@AD.UND.EDU

If you didn't follow the steps to configure the browser to attempt Kerberos auth with the site, the browser should pop-up a userid/password box, and if you enter the correct info, it should show the same info.

Conclusion

So there you have it, after 15 minutes of work you can now have a webpage tell you what your own AD userid is. OK, it's probably more useful than that - you can now write webapps proxied behind Apache, such as a Django app, that just have to look at the REMOTE_USER variable to tell who's on the other end of the connection.

You'll probably not want to require Kerberos auth for the whole app, but all you really need is to require Kerberos for one particular login URL that sets your userid into a session, and leave it up to the framework to check the session for authentication on the rest of the site.

IPv6 World Launch Day

IPv6 Logo

I've been working on getting this website up and running under IPv6, and it turned out to be somewhat involved. Firstly, I signed up with Hurricane Electric's tunnelbroker.net, to get IPv6 connectivity, because my ISP doesn't offer it yet. Setup my own DNS servers running nsd, which was a bit of a learning curve, but in the long run I think it'll be better than working with goofy DNS managers like you'd find on registrar or hosting websites.
NameCheap is now letting you setup IPv6 glue records right on their website (previously you had to file a support ticket), so that made things easier.

The only big glitch I ran into is that on FreeBSD, using simply

listen [::]:80;

to listen to both IPv4 and IPv6 didn't work. When trying that, I found that any request coming in as IPv4 would give weird 403 or 404 (I don't remember which) errors, where it seemed nginx just didn't know what virtual host to go to.
Linux doesn't seem to have that problem. Ended up using separate listen statements, as in:

listen 80 default_server;
listen [::]:80 default_server ipv6only=on;

for the main site, but VERY IMPORTANTLY, the remaining sites could not have the ipv6only=on directive, they just simply say

listen  80;
listen [::]:80;

(found that trick in this ServerFault page). This also has the advantage of showing proper IPv4 IP addresses in the logs, instead of IPv4-mapped IPv6 addresses such as ::ffff:11.22.33.44, so I ended up doing the same thing on a Linux box even though it handled dual-stack by default just fine.

I also for testing purposes, made aliases

To force one protocol or the other. When you use http://barryp.org/blog/, it's not obvious which you're using.

q2java and qwpython on GitHub

Quite some time ago I worked on a couple interesting projects, q2java which embedded Java into a Quake2 server and then allowed for games to be written in Java; and qwpython, which wrapped up the QuakeWorld dedicated server engine as a Python module and came with a QuakeC -> Python translator, so existing QW games like CTF could be converted to Python and hacked on from there.

Some time back I had created Subversion repositories of the various releases of those packages (didn't really know or use VCSes back then), and had them on an Apache mod_svn server. Well, SVN is kind of a PITA, and I'd like to not have to keep that server config going, so I looked at converting those repos to something else.

First tried hgsvn, which has worked pretty decently for smaller or simpler repos, but something in the q2java one make it crap out.

Next tried the hg convert extension, which worked better, but only when using a local SVN URL (like file://...), and then didn't manage to pick up the tags correctly.

Decided to give Git a try, first with Converting a Subversion repository to Git (in seven steps). It worked, but was kind of a nasty process for a Git newbie to wade through. The tags still didn't seem quite right.

Finally, realized GitHub has its own Subversion import built right into the website, and gave that a try. Very very nice, made importing the svn repo a breeze, and seems to have gotten the tags just right. Only took two steps: enter the svn URL, fill in a translation table of svn userids to git authors. I'd recommend this highly for anyone looking to move off svn. See the q2java and qwpython results here.

PyCon 2012

Headed off for PyCon 2012 tomorrow. Last one I was at was 2007 in Dallas, can't believe it's been 5 years. Looking forward to seeing some cool stuff, and maybe playing some games in the evening.

Debian GNU/kFreeBSD in a FreeBSD Jail - part 2

Previously I wrote about getting Debian GNU/kFreeBSD working in a jail. I've worked on it a bit more, polishing things up so I've got it working pretty seamlessly with my existing ezjail FreeBSD jails, so everything starts automatically, and you can use the ezjail commands to stop/restart the jail.

Here are a few more notes about how things got setup for my jail I named debian:

Kernel Modules

In /boot/loader.conf, I added these lines:

fdescfs_load="YES"
linprocfs_load="YES"
linsysfs_load="YES"
tmpfs_load="YES"

Mounting Filesystems

Created /etc/fstab.debian and populated with:

linproc     /jails/debian/proc      linprocfs       rw 0 0
linsys      /jails/debian/sys       linsysfs        rw 0 0
tmpfs       /jails/debian/lib/init/rw   tmpfs       rw 0 0

ezjail Config

Created /usr/local/etc/ezjail/debian with these contents:

export jail_debian_hostname="debian"
export jail_debian_ip="127.0.0.6"
export jail_debian_interface="lo0"
export jail_debian_rootdir="/jails/debian"
export jail_debian_mount_enable="YES"
export jail_debian_devfs_enable="YES"
export jail_debian_devfs_ruleset="devfsrules_jail"
export jail_debian_fdescfs_enable="YES"
export jail_debian_exec_start="/etc/init.d/rc 3"
export jail_debian_flags="-l -u root"

I also tried adding an IPv6 address to the jail, and that seems to work OK

So you can now stop/start with jail with

service ezjail.sh stop debian
service ezjail.sh start debian

Connect to the jail console

If you create a symlink for login (so that from the jail's POV there's a /usr/bin/login, like there would be on a FreeBSD jail)

cd /jails/debian/usr/bin/
ln -s ../../bin/login .

then you can use the ezjail-admin command to get a console in the jail, with:

ezjail-admin console debian

Otherwise, I've been using my own script to get a console (which assumes bash is installed in the jail), named /usr/local/sbin/jlogin

#!/bin/sh
#
# log into a jail, running bash
#
JID=`jls | grep " $1 " | awk '{print $1}'`
exec jexec $JID env -i PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin TERM=$TERM EDITOR=$EDITOR LANG=$LANG HOME=/root bash -l

That runs as:

jlogin debian

Debian GNU/kFreeBSD in a FreeBSD Jail

I've been a FreeBSD user for quite some time, going back to 3.3 or so, and for the last serveral years have also been working a lot with Ubuntu Linux. So when I ran across Debian GNU/kFreeBSD, which provides a Debian environment on top of a FreeBSD kernel, I was somewhat intrigued. It got even more interesting when I found a tutorial on setting up GNU/kFreeBSD in a jail. The notion of having a Debian environment on my home FreeBSD server without having to get something like VirtualBox running was just too good to pass up.

I got it running fairly decently, but along the way ran into some small problems - and thought I'd jot down what they were and what the fixes were.

FreeBSD Update

At first, I was using FreeBSD 8.2-RELEASE, and used debootstrap to install Debian Squeeze, as the tutorial showed. Once inside the jail, things sort of worked, but most commands, aptitude especially, would die with:

User defined signal 1

It turns out you need a newer kernel than 8.2 to run kFreeBSD in a chroot, as is mentioned in the FAQ. I upgraded my FreeBSD kernel/world to 8.3-PRERELEASE (2012-02-22), and the "signal 1" problem went away.

Debian Update

The next problem was that aptitude would still die, with:

Uncaught exception: Unable to read from stdin: Operation not permitted

After reading about this bug in cwidget, it seemed an upgrade to Wheezy was needed to fix the problem - and sure enough that problem went away.

kbdcontrol and /dev/console

The upgrade to Wheezy didn't go entirely smoothly, mainly due to the kbdcontrol package (required by sysvinit) being unable to access /dev/console in the jail. I wasn't worried about keeping things in the jail isolated for security reasons, so I went ahead and added /dev/console on-the-fly to the running jail by running outside the jail:

devfs -m /jails/debian/dev rule add path 'console*' unhide
devfs -m /jails/debian/dev rule applyset

After that, the kbdcontrol package was able to be upgraded, and I seem to have a Wheezy FreeBSD jail now. Very cool.

UPDATE: A followup talks more about the actual file changes made to run as an ezjail