Fun with ones and zeros - freebsd

I've been running lots of FreeBSD jails on various servers I maintain, and one thing I've noticed is that using ssh or scp from inside a jail often results in the error: Host key verification failed. A little Google searching turns up this explanation, that the problem is caused when you jexec into the jail instead of logging in normally through SSH.

I often run the jails in a pretty minimal way and don't really want to run sshd in them, and fortunately the problem can be worked around somewhat. Apparently the Host key verification failed. error is caused when SSH is unable to show you this type of prompt:

The authenticity of host 'foobar.edu' can't be established.
DSA key fingerprint is 7c:ac:b0:da:be:3c:c2:00:00:00:00:ce:db:fb:49:77.
Are you sure you want to continue connecting (yes/no)?

when connecting to a host you haven't connected to before. All you have to do to get around this is manually add a line to the jail's ~/.ssh/known_hosts for the server you're trying to connect to, probably by copying one from a known_hosts on another box or outside the jail.

Once past that, you may find that SSH is still unhappy in the jail if you don't have publickey authentication setup with the server you're trying to connect to, with an error like:

Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).

Fixing that is just a matter of generating or copying a private/public key pair into the jail's ~/.ssh directory, and putting the public key on the server you're connecting to.

posted at: 17:14 | tags: freebsd | 0 comments | permanent link to this entry

While working on my DiskCompare.com website I had a weird problem with daemontools. The Django process I had running under daemontools became unresponsive, it wouldn't shutdown normally with svc -d, and after kill -9 and restarting, it still wouldn't respond - it just seemed hung up or frozen. No messages in /var/log/messages, but running the service manually outside of daemontools worked fine. What the heck?

I had the service also setup to run multilog, and it turns out that I had mistakenly changed the ownership of the log directory it was supposed to be writing to. My guess is that it was the multilog process that was really hung up, since it couldn't write to the files it wanted to, and that my Django processes were then blocked because the pipe between them and multilog was full.

Simply correcting the permissions of the log directory cleared the jam and things took off again. So if you're seeing strange behavior with daemontools, that's one area to check out.

posted at: 16:01 | tags: daemontools freebsd | 0 comments | permanent link to this entry

In one of my FreeBSD 6.2 jails running Apache, even though the server seemed to respond ok, I saw lots of these errors in the logfile:

[warn] (61)Connection refused: connect to listener on 0.0.0.0:443

Google searching found lots of other people asking about this, but I didn't really see any good answers. Others complained about the same thing on port 80

[warn] (61)Connection refused: connect to listener on 0.0.0.0:80

I think the problem is just that Apache in a jail can't listen to :443 or 0.0.0.0:443 (or :80 or 0.0.0.0:80). If your jail has the IP 1.2.3.4 for example, then in httpd.conf, changing

Listen 80

to

Listen 1.2.3.4:80

and/or in extra/httpd-ssl.conf

Listen 443

to

Listen 1.2.3.4:443

Seems to fix the problem

posted at: 09:37 | tags: apache freebsd | 0 comments | permanent link to this entry

At work there are machines I'd like to access from home using Windows networking (Samba servers mostly), but the catch is that the work firewall is blocking NetBIOS traffic (an awfully good idea). My home network uses a FreeBSD box for NAT (Network Address Translation). Here's a diagram of what we're talking about.

In the picture above, my home NAT box has the public IP 1.2.3.4, the internal home network is 10.0.0.0/24, and I'm trying to reach a work server 5.6.7.100 on the 5.6.7.0/24 network.

Fortunately, I have a FreeBSD box at work, with an address 5.6.7.8, and it's fairly easy to setup a simple OpenVPN tunnel between 1.2.3.4 and 5.6.7.8, and route NetBIOS traffic over that.

OpenVPN will make it appear as if the two machines have a point-to-point network connection, when in reality the traffic is passing encrypted over the public internet. We need to pull a couple IP numbers out of our hat to use for the VPN endpoints - I'll use 192.168.88.1 for the home machine and 192.168.88.2 for the work machine.

Settting up OpenVPN

On each box install the security/openvpn port. After that's done, on one machine, go to /usr/local/etc/openvpn and run:

openvpn --genkey --secret mykey

Copy the mykey file you just generated over to the other box's /usr/local/etc/openvpn directory. The two OpenVPN endpoints will used that shared key to authenticate each other.

On the 1.2.3.4 machine, create a /usr/local/etc/openvpn/openvpn.conf file containing:

remote 5.6.7.8
dev tun0
ifconfig 192.168.88.1 192.168.88.2
secret mykey

On the 5.6.7.8 machine, create a /usr/local/etc/openvpn/openvpn.conf file containing:

remote 1.2.3.4
dev tun0
ifconfig 192.168.88.2 192.168.88.1
secret mykey

(note that the ifconfig line swapped the IPs compared to the other machine's config)

Throw an openvpn_enable="YES" in each machine's /etc/rc.conf, and start the daemons: /usr/local/etc/rc.d/openvpn start

If necessary, allow OpenVPN traffic through your firewall, for the 1.2.3.4 box it might look something like:

pass in on $ext_if inet proto udp from 5.6.7.8 to $ext_if port 1194
pass on tun0

If this works, you should be able to sit at the 1.2.3.4 box and ping 192.168.88.2 and get a response. On the 5.6.7.8 box, running tcpdump -n -i tun0 should show the ICMP packets reaching the machine.

Routing specific traffic

I don't want to route all my traffic going to the 5.6.7.0/24 network through the VPN, I just want the NetBIOS stuff so I'll setup a split tunnel. PF makes it pretty easy to redirect network traffic through the VPN, in fact, I ended up doing a double-NAT, one on each end of the tunnel.

So when the home workstation contacts the Samba server, the Samba server sees the traffic as coming from the 5.6.7.8 box, and the 5.6.7.8 box saw the traffic as coming from the home FreeBSD NAT machine. So interestingly, neither of the work machines needs to have any clue about the home network. The PF state tables take care of reversing everything when the Samba server responds.

On the home 1.2.3.4 machine, these lines are added in the appropriate places to /etc/pf.conf:

int_if="eth1"
internal_net="10.0.0.0/24"
work_net="5.6.7.0/24"
.
.
nat on tun0    from $internal_net to any -> (tun0)
.
.
pass in on $int_if route-to tun0 proto tcp from any to $work_net port {139, 445} flags S/SA modulate state

That last line is the key to the whole thing, it's responsible for diverting the traffic we want to go through the VPN instead of over the public internet. If you want to secure additional protocols, just add similar lines.

The PF config on the work 5.6.7.8 machine is simpler, just

nat on $ext_if from 192.168.88.1 to any -> 5.6.7.8

To perform that 2nd NATting, making VPN traffic seem like it came from the work box.

Lastly, both machine need gateway_enable="YES" in /etc/rc.conf. A home NAT box probably already has that though.

There's a lot more that OpenVPN can do, we barely scratched the surface with the simple setup described above, check the docs for more info.

posted at: 16:35 | tags: freebsd | 0 comments | permanent link to this entry

Earlier I wrote about DHCP failover, but there's another thing I thought I might mention that could be useful to others....

I had a problem in that one of my servers' CMOS clocks tends to be a bit off, maybe 90 seconds. When dhcpd starts up, it is unable to enter a normal failover state because of the time difference between it and the other dhcpd server.

I have

ntpdate_enable="YES"
ntpdate_flags="-b x.x.x.x"

in my /etc/rc.conf, along with running openntpd, but for some reason ntpdate wasn't setting the clock at boot time, and by the time openntpd got the clock tuned up, dhcpd had given up on trying to re-establish failover. Restarting dhcpd by hand later on always worked OK.

I think what was happening was that the network jack this server was plugged into wasn't coming alive quick enough to be up and running when ntpdate tried to do its thing. Something to do with the Cisco switch not having portfast enabled.

I don't have access to do anything about the switches, so I came up with the workaround of adding a simple script /usr/local/etc/rc.d/000.afterboot.sh to schedule a job to run a few minutes after the machine boots - to adjust the clock and restart dhcpd. It looks something like:

#!/bin/sh
at now + 5 minutes <<EOF
/etc/rc.d/ntpdate restart
/usr/local/etc/rc.d/isc-dhcpd restart
EOF

It's a bit of a kludge, but seems to do the trick.

posted at: 19:42 | tags: freebsd unix | 0 comments | permanent link to this entry

This is a revision of an earlier post which has instructions that no longer work.

Are you running an older version of FreeBSD, and getting errors like this when you try to build a port?

"/usr/ports/Mk/bsd.port.mk", line 2416: warning: String comparison operator should be either == or !=
"/usr/ports/Mk/bsd.port.mk", line 2416: warning: String comparison operator should be either == or !=
"/usr/ports/Mk/bsd.port.mk", line 2416: Malformed conditional (((${OSVERSION} < 504105 || (${OSVERSION} 
    >= 600000 && ${OSVERSION} < 600103) || (${OSVERSION} >= 700000 && ${OSVERSION} < 700012)) && 
    ${PKGORIGIN} != "ports-mgmt/pkg_install") || exists(${LOCALBASE}/sbin/pkg_info))

If so, it's because the ports maintainers have started using expressions in the ports Makefiles which are not understood by the versions of make that come with old FreeBSDs.

The official recommended fix would be to upgrade your FreeBSD, but if that's not practical you can at least install a newer version of make to get by for a bit longer. This can be done in just a few minutes with two main steps: temporarily bring back an older /usr/ports/Mk which is compatible with FreeBSD 4.x - and then build and install the devel/make port which used to be present in the ports tree.

The ports tree is in CVS, so it's possible to checkout older revisions of selected directories. This FreeBSD Handbook page lists the anonymous CVS repositories available. For this example I'm going to use anoncvs@anoncvs1.FreeBSD.org:/home/ncvs We only need two small directories, so it probably doesn't really matter which one you use.

It seems like the first commit to the ports infrastructure which broke compatibility happened around Feb 5th, 2007 - so let's backup the current /usr/ports/Mk and check out one from Feb 4th:

cd /usr/ports
mv Mk Mk.original
cvs -d anoncvs@anoncvs1.FreeBSD.org:/home/ncvs co -D "04 Feb 2007" -d Mk ports/Mk

You should now be able to build some ports, at least those that don't use incompatible syntax in their individual Makefiles and don't require a newer ports infrastructure. Now, if you don't already have a devel/make port, use CVS to bring that back too, then build and install it:

cd /usr/ports/devel
cvs -d anoncvs@anoncvs1.FreeBSD.org:/home/ncvs co -D "04 Feb 2007" -d make ports/devel/make
cd make
make install
make clean

Lastly, set your system to use the new ports make in place of the old system make, and do some cleanup:

cd /usr/bin
mv make make.old
ln -s /usr/local/bin/make .

cd /usr/ports
rm -rf Mk
mv Mk.original Mk
rm -rf /usr/ports/devel/make

You should now be in better shape for trying to build new ports. 4.x isn't officially supported anymore by the ports maintainers, so there may be some individual port breakage - but at least you're over the first hurdle.

posted at: 14:41 | tags: freebsd | 0 comments | permanent link to this entry

Just a followup to my earlier post on updating FreeBSD to handle the changes in Daylight Savings Time starting in 2007....

There are a couple mentions [1], [2] on the freebsd-stable mailing list that the /etc/localtime file is binary compatible across various versions of FreeBSD.

This means you only have to get one good copy of /etc/localtime for your timezone, either by pulling from a FreeBSD 6.2 or higher machine, or installing the misc/zoneinfo port on one older machine and run tzsetup. Once you have that good version, you can copy it around to your other FreeBSD boxes, regardless of what version they're running.

This seems to also work between FreeBSD and OpenBSD (and perhaps Net and Dragonfly). I had a single OpenBSD 3.7 machine I wanted to update, and it seems to work OK to use a /etc/localtime pulled from a FreeBSD box. On OpenBSD (3.7 at least), /etc/localtime was a symlink to a file in /usr/share/zoneinfo/. I just removed the symlink and made a new one pointing to a new timezone file I stuck somewhere on the disk. Checking for the correct DST dates in OpenBSD seems to be the same as with FreeBSD:

zdump -v /etc/localtime | grep 2007

Look for March 11th as the start date.

posted at: 13:37 | tags: freebsd openbsd | 0 comments | permanent link to this entry

The instructions in this entry no longer work, please check out the revised entry on this issue.

I've got a couple older FreeBSD machines at work, running 4.7 and 4.8, that have started having trouble building ports. Apparently some changes have been made to the ports infrastructure that are not compatible with make on older FreeBSDs. The error message given when a running 'make' in a port directory is:

"/usr/ports/Mk/bsd.port.mk", line 2292: warning: String comparison operator should be either == or !=
"/usr/ports/Mk/bsd.port.mk", line 2292: warning: String comparison operator should be either == or !=
"/usr/ports/Mk/bsd.port.mk", line 2292: Malformed conditional (((${OSVERSION} < 504105 || 
      (${OSVERSION} >= 600000 && ${OSVERSION} < 600103) 
      || (${OSVERSION} >= 700000 && ${OSVERSION}     < 700012)) 
     && ${PKGORIGIN} != "ports-mgmt/pkg_install") || exists(${LOCALBASE}/sbin/pkg_info))
"/usr/ports/Mk/bsd.port.mk", line 2293: warning: String comparison operator should be either == or !=
"/usr/ports/Mk/bsd.port.mk", line 2293: warning: String comparison operator should be either == or !=
"/usr/ports/Mk/bsd.port.mk", line 2293: Malformed conditional ((${OSVERSION} < 504105 || 
     (${OSVERSION} >= 600000 && ${OSVERSION} < 600103) 
     || (${OSVERSION} >= 700000 && ${OSVERSION} < 700012)) 
     && ${PKGORIGIN} != "ports-mgmt/pkg_install")
"/usr/ports/Mk/bsd.port.mk", line 2308: if-less else
"/usr/ports/Mk/bsd.port.mk", line 2308: Need an operator
"/usr/ports/Mk/bsd.port.mk", line 2322: if-less endif
"/usr/ports/Mk/bsd.port.mk", line 2322: Need an operator
"/usr/ports/Mk/bsd.port.mk", line 5987: if-less endif
"/usr/ports/Mk/bsd.port.mk", line 5987: Need an operator
make: fatal errors encountered -- cannot continue

I ran into this when trying to update timezone info. The problem can be fixed by installing a newer version of make on the system.

First, backup /usr/ports/Mk/bsd.port.mk

 cd /usr/ports/Mk
 cp -p bsd.port.mk bsd.port.mk.orig

Then patch it with this file or edit with your favorite editor going to line 2292 (the first error line above), where there's a chunk of .if .if....endif .endif statements and tossing everything except this first cluster of variables being set:

PKG_CMD?=              ${LOCALBASE_REL}/sbin/pkg_create
PKG_ADD?=              ${LOCALBASE_REL}/sbin/pkg_add
PKG_DELETE?=   ${LOCALBASE_REL}/sbin/pkg_delete
PKG_INFO?=             ${LOCALBASE_REL}/sbin/pkg_info
PKG_VERSION?=          ${LOCALBASE_REL}/sbin/pkg_version

save it. Now build and install a newer version of make, "Berkeley make, back-ported to FreeBSD 4.x"

cd /usr/ports/devel/make
make install
make clean

use it instead of the old make

cd /usr/bin
mv make make.old
ln -s /usr/local/bin/make .

restore the bsd.ports.mk file you backed up

cd /usr/ports/Mk
mv bsd.port.mk.orig bsd.port.mk

You should be able to build ports again using the current ports tree infrastructure.

posted at: 16:03 | tags: freebsd | 0 comments | permanent link to this entry

Starting in 2007, the dates that daylight savings time begins and ends is changing in the US and other countries. For FreeBSD it seems versions 6.2 and higher should already know about the new DST dates. A machine can be checked with

zdump -v /etc/localtime | grep 2007

A machine with the old DST settings will show lines that begin with:

/etc/localtime  Sun Apr  1 07:59:59 2007 
/etc/localtime  Sun Apr  1 08:00:00 2007 
/etc/localtime  Sun Oct 28 06:59:59 2007 
/etc/localtime  Sun Oct 28 07:00:00 2007

which is wrong, (April 1st and Oct 28th). A machine that's correct should show:

/etc/localtime  Sun Mar 11 07:59:59 2007 
/etc/localtime  Sun Mar 11 08:00:00 2007 
/etc/localtime  Sun Nov  4 06:59:59 2007 
/etc/localtime  Sun Nov  4 07:00:00 2007

March 11th and Nov 4th being the new days that DST switches.

Updating a FreeBSD box seems to be just a matter in installing the misc/zoneinfo port, and then running the tzsetup command which gives you menus to pick your timezone again.

this blog has links for info on updating other OSes.

posted at: 10:57 | tags: freebsd | 0 comments | permanent link to this entry

I'd like to add a search feature back on this site. Previously, I had an arrangement setup with PyBlosxom and ht://Dig, but now that it's Django-powered, I'd like to do something that worked directly from the database instead of crawling the site like ht://Dig did.

After looking a while at various text search engines, I remembered seeing that SQLite just added an FTS1 module in version 3.3.8, which sounds pretty easy to use. Unfortunately the FreeBSD port databases/sqlite3 doesn't build with that feature.

After poking around a bit, I got it to build with FTS manually, and after a whole bunch more messing around, came up with a patch to add an option to the port to build sqlite3 with FTS. The patch has been submitted to the FreeBSD bug tracker as ports/106281. Hopefully I have all my ducks lined up on that.

Anyhow, in testing the FTS a bit, I found one thing they only hint at on the SQLite website. There's a Porter stemmer built in, even though the wiki says: "The module does not perform stemming of any sort." You activate it by adding tokenize porter to the table declaration, for example (adapted from their example).

CREATE VIRTUAL TABLE recipe USING FTS1(name, ingredients, tokenize porter);

once you've done that, and inserted some sample data:

INSERT INTO recipe VALUES('broccoli stew', 'broccoli peppers cheese tomatoes');

the searches don't have to be quite as exact, for example:

SELECT name FROM recipe WHERE ingredients MATCH 'pepper'

hits the 'broccoli stew' recipe even through it has 'peppers' and you searched for 'pepper'.

Not sure why the Porter stemmer isn't documented in the SQLite wiki, perhaps it's still a work in progress or being changed for FTS2.

posted at: 18:20 | tags: bugs freebsd sqlite | 0 comments | permanent link to this entry