General Purpose App Server

Hardware

| Type | Make | Model | Connector | Specs | || | Motherboard | | Xeon | Dual Xeon Slot | | | CPU | Intel | Pentium III Xeon | Slot | 550MHz | | CPU | Intel | Pentium III Xeon | Slot | 550MHz | | RAM | Kingston | | | 256MB ECC | | HDD | IBM | | SCSI | 9GB | | HDD | IBM | | SCSI | 9GB | | Graphics Adapter | Diamond | FireGL 1000 Pro 8MB AGP | AGP | 8MB 100MHz SGRAM 8ns | | SCSI Controller | Adaptec | AIC-7880 | PCI? | | | SCSI Raid Adapter | Adaptec | ARO-1130C | PCI? | 16MB | | CD-ROM | NEC | | SCSI |

Plan

Create jails for mail, www and (probably) dns

ToDo

Get fan for CPUs
http://memberwebs.com/nielsen/freebsd/jails/docs/jail_security.html
create CA and sign certs
create proper file permission hierarchy for cert access
set soft_bounce in main.cf to NO
add local net back to mynetworks in main.cf
remove ALL unused accounts
run dovecot-auth as user dovecot-auth instead of root
set dovecot first_valid_uid & last_valid_uid to dovecot user UID
set dovecot listen to localhost only. disallow remote non-SSL imap

Log

Earlier

Check Adaptec RAID support
create generic kernel
serial console

/etc/ttys
    options  CONSPEED=115200

boot console

/boot.config: -P
    /boot/loader.conf: console="comconsole"

create custom kernel

build custom boot blocks

setup panic debugging

minor change: KDB_UNATTENDED
    set dumpdev
    set savecore_flags="-z"

jail config

reconfigure daemons

add dual PIII 550

breaking off heatsink fins

build SMP kernel

options SMP
    device apic

2006-04-12

Updated to FreeBSD 6-STABLE branch. http://www.freebsd.org/doc/en/books/handbook/makeworld.html

Fill in: CVSup config, make.conf

# cd /usr/src
# make -j4 buildworld && make buildkernel

Ran buildworld overnight.

2006-04-13

Powered itself off overnight. Don't know why.
Finished upgrade to 6-STABLE
Rebooted, SMP working
Started creation of jails
http://www.onlamp.com/pub/a/bsd/2003/09/04/jails.html

# export JAILDIR=/chroot/test
# cd /chroot
# mkdir test mail
# cd /usr/src
# make -j6 buildworld
# make installworld DESTDIR=$JAILDIR

http://www.section6.net/wiki/index.php/Creating_a_FreeBSD_Jail

# make world DESTDIR=/chroot/test
# sockstat|grep "\*:[0-9]"
# vim /etc/rc.conf
syslogd_flags="-s -s"
syslogd_flags="-a $local_ip"

Beeped lots then powered off
Burnt finger on CPU heatsinks. OK. I get it now.
No CPU fan, dual PIII definitely needs one.

2006-04-14

http://www.section6.net/wiki/index.php/Creating_a_FreeBSD_Jail

Created minimal make.conf.jail

# cd /usr/local/etc
# cp /etc/make.conf make.conf.jail
# vim make.conf.jail
...disable lots of things...

I disabled everything except for:

CRYPT
DYNAMICROOT
INET6
LIBC_R
LIBPTHREAD
LIBTHR
OPENSSL
SHARE
SHARED

This is a production jail, so no toolchain, no docs. Only the bare essentials.

Backup host make.conf and copy the jail one over it.

# cp /etc/make.conf /etc/make.conf.bak
# cp /usr/local/etc/make.conf.jail /etc/make.conf

Remember to copy it back afterwards.

Clean, then buildworld:

# cd /usr/obj
# chflags -R noschg *
# rm -rf *
# cd /usr/src
# make -j4 buildworld

Meanwhile, still following section6wiki:

# cd $JAILDIR
# mkdir etc dev usr
# cp /etc/resolv.conf etc/resolv.conf
# cp /etc/rc.conf etc/rc.conf
# vim rc.conf
...edit stuff, hostname, ip...
# mount_devfs devfs $JAILDIR/dev # a bit preemptive perhaps?
# devfs -m $JAILDIR/dev rule -s 4 applyset
# ln -s dev/null kernel

Build finished so install:

# cd /usr/src
# make installworld DESTDIR=$JAILDIR
# cd etc
# make distribution DESTDIR=$JAILDIR NO_OPENSSH=YES
# cd $JAILDIR
# touch etc/fstab

Create alias manually for now. It's added to rc.conf further on.

# ifconfig xl0 $jail_ip netmask 255.255.255.0 alias

Restore make.conf, append jail config to rc.conf:

# cd /etc
# cp make.conf.bak make.conf
# cat >> rc.conf
jail_enable="YES"
jail_list="test"
jail_set_hostname_allow="NO"
jail_socket_unixiproute_only="YES"
jail_test_rootdir="/chroot/test"
jail_test_hostname="test.niggurath.local"
jail_test_ip="$jail_ip"
jail_test_exec_start="/bin/sh /etc/rc"
jail_test_devfs_enable="YES"
jail_test_devfs_ruleset="devfsrules_jail"
ifconfig_xl0_alias0="inet $jail_ip netmask 255.255.255.0"
^D
#

Quick test run:

# jail $JAILDIR test.niggurath.local $jail_ip /bin/sh /etc/rc
# jexec $JID /bin/sh

2006-04-16

Create postfix and dependent packages (See [Mail][7]):

http://www.onlamp.com/pub/a/bsd/2003/08/07/FreeBSD_Basics.html

# cd /usr/ports/mail/postfix
# make
... select options...
(enabled BerkeleyDB (latest), TLS, LDAP, SPF)
# mkdir /usr/ports/packages
# make package-recursive
...answer questions...

I *might* use LDAP to store user accounts and authentication details on the network. We'll see.

For some reason I can't mount nullfs or unionfs on Niggurath. Perhaps something to do with gmirror? Haha, retard. securelevel of host was 1, so the nullfs kernel module couldn't be loaded.

Change Niggurath host securelevel to -1 AND autoload nullfs.ko:

# cat >> /etc/rc.conf
kern_securelevel="-1"
kern_securelevel_enable="YES"
^D
# echo 'nullfs_load="YES"' >> /boot/loader.conf

Reboot, or shutdown/exit or whatever in order to change the securelevel

Now load ports over the jail ports directory and install postfix:

# mount_nullfs /usr/ports $JAILDIR/usr/ports
# jexec 1 /usr/local/bin/bash
[#] cd /usr/ports/packages/mail
[#] pkg_add ./postfix-2.2.5,1.tbz
You need user "postfix" added to group "mail".
Would you like me to add it [y]? y
Would you like to activate Postfix in /etc/mail/mailer.conf [n]? y

http://www.csua.berkeley.edu/~ranga/notes/freebsd_postfix.html

Set in $JAILDIR/etc/rc.conf:

sendmail_enable="YES"
sendmail_flags="-bd"
sendmail_pidfile="/var/spool/postfix/pid/master.pid"
sendmail_procname="/usr/local/libexec/postfix/master"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"
sendmail_msp_queue_enable="NO"

2006-04-17

http://www.postfix.org/INSTALL.html#replace
(Note that config is in $JAILDIR/usr/local/etc/postfix)

Set domain names for all servers to *.phase1 to make it easy to grep.

# vim $JAILDIR/etc/rc.conf
jail_test2_hostname="mail.phase1"

Other files changed: /etc/hosts /etc/resolv.conf

Change the jail ip address.
Fix interface alias:

# jexec $JAIL_ID kill -TERM -1
# ifconfig xl0 -alias $jail_ip
# ifconfig xl0 alias $new_jail_ip netmask 255.255.255.0
# export jail_ip=$new_jail_ip

Restart the jail:

#jail $JAILDIR mail.phase1 $jail_ip /bin/sh /etc/rc

Now back to postfix configuration.
The original configuration didn't start postfix. New rc config options are here:
http://www.freebsddiary.org/postfix.php

Set in $JAILDIR/etc/rc.conf:

postfix_enable="YES"
sendmail_enable="NO"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"
sendmail_msp_queue_enable="NO"

Created and updated periodic.conf

# cat >> $JAILDIR/etc/periodic.conf
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"
^D

No mailwrapper, so fix the sendmail link and link the startup script
http://www.csua.berkeley.edu/~ranga/notes/freebsd_postfix.html

# cd $JAILDIR/usr/sbin
# ln -s ../local/sbin/sendmail sendmail
# cd $JAILDIR/usr/local/etc/rc.d
# ln -s ../../sbin/postfix postfix.sh

Forgot to generate /etc/aliases.db:

[#] /usr/local/bin/newaliases

2006-04-19

Gah. Ok, I should have built Postfix with SASL. (Not much point in having encryption (TLS) without no authentication (SASL) to protect.

No special build steps should be necessary. Maaaybe cyrus-sasl2-ldapdb. But that can be installed later. To rebuild the Postfix port, including SASL this time:

# cd /usr/ports/mail/postfix
# make clean
# make
...this time select
SASL2
TLS
DB43 (just in case. so I have a fallback if LDAP is too much of a pain)
OpenLDAP
VDA (it has quotas)

Once compiled you may have to uninstall existing packages with pkg_delete.

Build the updated packages:

# cd /usr/ports/mail/postfix
# make package-recursive
# cd /usr/ports/security/cyrus-sasl2-saslauthd/
# make package-recursive

Install in the jail:

# jexec $JID /bin/sh
[#] pkg_delete postfix-2.2.5,1
[#] cd /usr/ports/packages/All
[#] pkg_add postfix-2.2.5,1.tbz
[#] pkg_add cyrus-sasl2-saslauthd

Back to configuration.
http://wanderingbarque.com/howtos/mailserver/mailserver.html

Create the vmail user, with home at /vmail and no login shell

[#] pw useradd vmail -m -d /vmail -s /usr/sbin/nologin
[#] cd /vmail
[#] rm -rf .*
[#] mkdir domains
[#] chown vmail:vmail domains

No particular reason for choosing /vmail. It's just simple.

http://yocum.org/faqs/postfix-tls-sasl.html

# cd $JAILDIR/usr/local/lib/sasl2/
# echo "pwcheck_method: saslauthd" > smtpd.conf
# mkdir deactivated
# mv *ntlm* *crammd5* deactivated

Create cert for mail.phase1
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/openssl.html

# mkdir $JAILDIR/etc/certs
# cd $JAILDIR/etc/certs
# openssl req -new -x509 -nodes -out mailkey.pem -keyout mailcert.pem -days 3650
# cd ..
# chmod -R 700 certs




# cat >> $JAILDIR/usr/local/etc/postfix/main.cf

# SASL authentication - added by root - 2006/04/19
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain =

smtpd_sender_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination

# tls config
smtp_use_tls = yes
smtpd_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtpd_tls_key_file = /etc/certs/keycert.pem
smtpd_tls_cert_file = /etc/certs/keycert.pem
smtpd_tls_CAfile = /etc/certs/keycert.pem
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

saslauthd failed with unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so: Shared object "libgssapi.so.8" not found, required by "libgssapiv2.so"

Have to rebuild without Kerberos this time.

# jexec $JID /usr/sbin/pkg_delete cyrus-sasl2-saslauthd-2.1.21
# cd /usr/ports/security/cyrus-sasl2-saslauthd/
# make WITH_OPENLDAP_VER=22 WITHOUT_GSSAPI= package
# jexec $JID /bin/sh
[#] /usr/sbin/pkg_add cyrus-sasl2-saslauthd-2.1.21
[#] saslauthd -a getpwent
[#] postfix reload

getpwent will be replaced with LDAP once implemented.

# vim $JAILDIR/usr/local/etc/postfix/master.cf
...uncomment...
submission   inet    n       -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_enforce_tls=yes -o smtpd_sasl_auth_enable=yes@@

2006-04-20

I updated ports with cvsup and updated postfix:

# cd /usr/ports/mail/postfix
# make package
# make deinstall && make reinstall
# jexec $JID /bin/sh
[#] cd /usr/ports/packages/All
[#] pkg_delete postfix-2.2.5,1
[#] pkg_add postfix-2.2.10_1,1.tbz

Installing Dovecot (1.0.beta3):

# cd /usr/ports/mail/dovecot
# make package-recursive
...select
KQUEUE
SASL2
LDAP

GnuTLS was removed from Dovecot as it was broken. Falls back to OpenSSL instead. Fine by me. Auto-creates user and group dovecot.

Then it broke. I think because older versions of libiconv and cyrus-sasl were already installed (by postfix). So build just the dovecot package:

# make package

Then it worked.

[#] pkg_add /usr/ports/packages/All/dovecot-1.0.b3_1.tbz

Add dovecot-auth user and group

[#] pw useradd dovecot-auth -n "Dovecot Auth" -m -d /usr/local/libexec/dovecot -s /usr/sbin/nologin

Actually this produced the error: pw: invalid character ` ' at position 7 in userid/group name, so I added the user with sysinstall...

Configure Dovecot. Enable imap and imaps. imap accepts connections from the local network, imaps from anywhere. The firewall must protect from outside access to imap.

# cd $JAILDIR/usr/local/etc
# cp dovecot-example.conf dovecot.conf
# vim dovecot.conf

dovecot.conf:

protocols = imap imaps
# must be guarded by firewall!
listen = *
ssl_listen = *
ssl_disable = no
ssl_cert_file = /etc/certs/mailcert.pem
ssl_key_file = /etc/certs/mailkey.pem
login_user = dovecot
login_process_size = 64
login_max_processes_count = 32
login_greeting = Welcome.
max_mail_processes = 128
verbose_proctitle = yes
# allow test user and dovecot
first_valid_uid = 1002
last_valid_uid = 1003
# allow test user group and dovecot group
first_valid_gid = 1002
last_valid_gid = 1003
mail_extra_groups = mail
valid_chroot_dirs = /vmail/domains
default_mail_env = maildir:/vmail/domains/%d/%n
# no mbox (thunderbird), no OSX Mail or Outlook Express
protocol imap {
  imap_client_workarounds = outlook-idle netscape-eoh
}
auth default {
  # plain for testing only. can't use digest-md5 with pam
  mechanisms = plain
  # for testing purposes, until LDAP is working
  passdb pam {
  }
  userdb passwd {
  }
  user = root
}

Enable dovecot at boot:

# cat > $JAILDIR/etc/rc.conf
dovecot_enable="YES"
^D

OK. Well, no quota support. Perhaps dump dovecot in favour of Courier...

2006-04-22

Now on to the DNS/LDAP server!

Create new jail (I'm so lazy):

# export JD=/chroot/dns
# export JIP=$jail_ip
# cp -Rp /chroot/mail $JD
# rm -rf $JD/dev  # oops!
# vim $JD/etc/rc.conf
...leave the sendmail disable stuff there
# mount_nullfs /usr/ports $JD/usr/ports
# mount_devfs devfs $JD/dev
# devfs -m $JD/dev rule -s 4 applyset
# ifconfig xl0 alias $JIP netmask 255.255.255.0
# jail $JD dns.phase1 $JIP /bin/sh /etc/rc
# jexec $JID pkg_delete -a   # delete all
#   # clean up the rest manually

Now install djbdns:

# cd /usr/ports/dns/djbdns
# vim Makefile
...comment out the NO_PACKAGE line...
# vim /usr/ports/sysutils/ucspi-tcp/Makefile
...comment out the NO_PACKAGE line...
# vim /usr/ports/sysutils/daemontools/Makefile

This is just an annoyance, we're creating packes to install in our own damn jail, ffs!

# make config-recursive
# make package-recursive
...select options
IPV6 # sugar-coating to make IPv6 records easier to specify

I always prefer to run the config step separately.

2006-04-24

Back to dovecot for a moment. Forgot to enable it on boot:

# export JD=/chroot/mail
# cat >> $JD/etc/rc.conf
dovecot_enable="YES"
saslauthd_enable="YES"
^D

Now on with djbdns. Install and configure.
http://www.lifewithdjbdns.com/#Setting%20up%20tinydns

[#] jexec $JID /bin/sh
[#] pkg_add /usr/ports/packages/All/djbdns-ipv6-1.05.b14_10.tbz
[#] pw useradd tinydns -s /usr/sbin/nologin -c "TinyDNS Owner"
[#] pw useradd dnslog -s /usr/sbin/nologin -c "DNS Log owner"
[#] tinydns-conf tinydns dnslog /usr/local/etc/tinydns $JIP
[#] mkdir -p /var/log/tinydns
[#] chown dnslog:dnslog /var/log/tinydns

Set the log directory in /usr/local/etc/tinydns/log/run

# vim $JD/usr/local/etc/tinydns/log/run
:s%./main%/var/log/tinydns%
:wq

Configure DNS data and start tinydns

# cat > $JD/usr/local/etc/tinydns/root/data

#define the authoritative nameserver
.phase1::ns1.phase1
#mail exchanger
@phase1::mail.phase1
=mail.phase1:192.168.1.203
#dns1 is also known as ns1
=dns1.phase1:192.168.1.204
+ns1.phase1:192.168.1.204
=www.phase1:192.168.1.205
^D

Compile data and start tinydns

[#] cd /usr/local/etc/tinydns/root
[#] make
[#] mkdir -p /var/service
[#] ln -s ../../usr/local/etc/tinydns /var/service

Use relative links so they work in and outside the jail.

Start daemontools at boot:
http://matt.simerson.net/computing/dns/djbdns-freebsd.shtml

# cat >> $JD/etc/rc.conf
svscan_enable="YES"

2006-04-25

Configure tinydns so prevent lame DNS lookups. This is very important!

# cat > $JD/usr/local/etc/tinydns/root/data
# RFC1918 - prevent pollution
.local::ns1.phase1   # not RFC1918, but useful anyway
.10.in-addr.arpa::ns1.phase1
.168.192.in-addr.arpa::ns1.phase1
.16.172.in-addr.arpa::ns1.phase1
.17.172.in-addr.arpa::ns1.phase1
.18.172.in-addr.arpa::ns1.phase1
.19.172.in-addr.arpa::ns1.phase1
.20.172.in-addr.arpa::ns1.phase1
.21.172.in-addr.arpa::ns1.phase1
.22.172.in-addr.arpa::ns1.phase1
.23.172.in-addr.arpa::ns1.phase1
.24.172.in-addr.arpa::ns1.phase1
.25.172.in-addr.arpa::ns1.phase1
.26.172.in-addr.arpa::ns1.phase1
.27.172.in-addr.arpa::ns1.phase1
.28.172.in-addr.arpa::ns1.phase1
.29.172.in-addr.arpa::ns1.phase1
.30.172.in-addr.arpa::ns1.phase1
.31.172.in-addr.arpa::ns1.phase1
^D

This makes tinydns authoritative for all RFC1918 addresses (private IP blocks). Setting up dnscache to consult tinydns for these IP blocks then prevents reverse-lookups being leaked to the internet.

It's fine to have these globally visible because there shouldn't be any DNS records (A, AAAA, MX or whatever) for these IPs in a public auth DNS. If there are, remote clients can get information about hidden hosts.

Setting up dnscach to consult the auth DNS for RFC1918 addresses:
Creating a file containing the IP of the auth DNS for each block:

# cd /var/service/dnscache/root/servers
# echo "192.168.1.204" > local
# cp local 168.192.in-addr.arpa
# cp local 10.in-addr.arpa::ns1.phase1
# for i in 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
> do cp 168.192.in-addr.arpa $i.172.in-addr.arpa
> done
#

Niggurath