Featured

zimbra logs behind reverse proxy

Zimbra is a beast; so when something goes awry, it can be a real pain fixing it. So, in this case we had a reverse proxy (11.12.13.14) serving tens of sites, serving among them a zimbra installation on mail.xxxx.yyy. The zimbra installation had of course a different host name, zimbra.xxxx.yyy and two IP addresses (21.22.23.24 & 31.32.33.34). In the /opt/zimbra/log/audit.log they wanted to see not the reverse proxy server’s IP address, but the connecting client’s IP address. There is a very nice guide on how to accomplish this in the zimbra documentation, but after following it to the t, we still were seeing all the connections and logins coming from 21.22.23.24. Not happy. What the guide fails to say, is this:

You need to add all the aforementioned IP address to the specific server’s zimbraMailTrustedIP

.
That is, for each server zmprov gas returns, you need to explicitly add the IP address listed in the audit.log (21.22.23.24 in this case) as a zimbraMailTrustedIP like this:
$ zmprov gas
zimbra.xxxx.yyy
$ zmprov ms zimbra.xxxx.yyy +zimbraMailTrustedIP 21.22.23.24

And then restart mailboxd with zmmailboxdctl restart. And then and only then will you see the real IP of the clients listed in audit.log, mailbox.log, etc.

Home -or office- xDSL router with a PCEngines APU3 and OpenBSD

I love PCEngine’s boards, so I always have a couple lying around. In fact an APU with an AMD G-T40E CPU and 4G RAM running Linux is my home router for the past 6 years. I always had a mind to replace it with an OpenBSD one, so last Saturday I started building one, and after some trial and error, this is how it worked. And to the obvious question of why would a sane person insert an extra router to a home network, features, I say.

 

The network diagram is simple; it’s a home network after all. An ADSL modem/router with some ethernet interfaces, one of them connected to the APU. The APU connects to the internet via the modem using pppoe and all other devices either connect to the APU using wireless or via an ethernet switch.

What needs to happen goes like this: the internet connection needs to be started and some firewall and/or routing rules need to be be applied; a DHCP server will give out IP addresses to the wired and wireless clients; a wireless access point need also be configured; a DNS server will be good to have, but not imperative -the clients can always be configured to query an external DNS. Many ISPs offer IPv6 connectivity too and the clients should be presented at least with such an offer. Something needs to monitor the internet connection and try to reconnect if needed.

The APU board provides 3 ethernet interfaces (em0 -next to the USB ports-, em1, em2) and can be fitted with a wireless board. The one I have came with an Atheros chip, and OpenBSD presents it as athn0. What will happen is, em0 and athn0 will be bridged in bridge0. In OpenBSD a bridge cannot be assigned an IP address, so a vether interface, vether0, will join the bridge and be configured with an IP address (192.168.1.5/24). The middle interface, em1, will be connected with the ADSL modem and will be configured dynamically with pppoe.

First, the simple things. A pppoe interface (pppoe0) connecting ethernet port em0 of the APU to an ethernet port of the xDSL router. Create /etc/hostname.pppoe0 and put the following in it:

inet 0.0.0.0 255.255.255.255 NONE \
pppoedev em1 authproto chap \
authname 'my_isp_user_name' authkey 'my_isp_password' up
dest 0.0.0.1
inet6 autoconf
!/sbin/route add default -ifp pppoe0 0.0.0.1
!/sbin/route add -inet6 default -ifp pppoe0 fe80::%pppoe0
!/sbin/pfctl -f /etc/pf.conf

Create /etc/pf.conf and put the following in it:

ext_if="pppoe0"
int_if="vether0"
icmptypes="echoreq"
table persist
table persist
table persist
table persist
#scrub in
set skip on lo
block return # block stateless traffic
pass # establish keep-state
match out on $ext_if inet from $int_if/24 to any nat-to $ext_if
#pass out on $ext_if inet6 from $int_if to any
pass out on $ext_if inet6
#pass out on $ext_if from 192.168.1.0/24 to any nat-to $ext_if
pass in on $int_if from $int_if/24
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
pass in on $ext_if inet6 proto icmp6 all icmp6-type { routeradv neighbrsol neighbradv }
pass in on $ext_if inet6 proto udp from fe80::/10 port dhcpv6-server to fe80::/10 port dhcpv6-client no state

Internet connectivity has not been established yet and even if there had been, I have not defined any name servers. So, edit or create /etc/resolve.conf and put a name server or two in it:

nameserver 8.8.8.8
nameserver 1.1.1.1

Now a basic level of functionality should be available, if only the board connects to the internet. Time to type:

sh /etc/netstart pppoe0

and hopefully a new interface should be created named pppoe0 and an IP address should be assigned to it. A good start.

Some extra packages will be needed, so now that internet is available, it 's a good time to go get them:

pkg_add dhcpcd dnsmasq

dhcpcd is needed only if IPv6 connectivity is desired. Otherwise, dnsmasq should suffice. If dhcpcd is not installed, the line !/etc/rc.d/dhcpcd restart from /etc/hostname.pppoe0 should be removed. Having said that, if you still wish to have IPv6 addresses in your LAN, create /etc/dhcpcd.conf and put the following in it:

hostname bbj.room.mikroskosmos.gr
allowinterfaces pppoe0
interface pppoe0
ipv6only
ia_pd 1 vether0
duid
persistent
vendorclassid
option domain_name_servers, domain_name, domain_search
option classless_static_routes
option interface_mtu
option host_name
option rapid_commit
require dhcp_server_identifier
slaac private

If IPv6 is desired, the rad daemon should also be started.

The board should have internet by now, but no clients can connect yet. An SSID, a DHCP server and dnsmasq must be configured before any clients can connect. DHCP first. Create /etc/dhcpd.conf and put something along these lines in it:

authoritative;
default-lease-time 36000; # 1 hour
max-lease-time 432000; # 12 hours
option domain-name "room.mikroskosmos.gr";
option routers 192.168.1.5;
option domain-name-servers 192.168.1.5;
option broadcast-address 192.168.1.255;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.40 192.168.1.59;
option routers 192.168.1.5;
}

A bridge (bridge0) with an ethernet interface (em1) and the wireless interface (athn0). Create "/etc/hostname.bridge0" and put the following in it:

add vether0
add em0
add athn0
up
!/etc/rc.d/dnsmasq restart

Time to configure the wireless interface. Create /etc/hostname.athn0 with the following lines:

nwid megaloskosmos
chan 9
mode 11g
mediaopt hostap
wpa
wpakey 4140280137537016
wpaprotos wpa2
up

Then create /etc/hostname.vether0 and add the following lines:

inet 192.168.1.5/24
inet6 autoconf

And lastly create /etc/hostname.em0 with just the word up in it:

echo up > /etc/hostname.em0

At this point the bridge interface should be configured and ready to be started:

sh /etc/netstart bridge0

Abusing fail2ban to train rspamd

Some time ago I enabled a more verbose logging in dovecot and noticed messages moved between INBOX and Junk. That came as a surprise, because I always thought people deleted junk messages. Turns out, they take the pain to move them in the appropriate folder, believing server will learn from this and take appropriate action in the future. This not the case with most smaller providers. Sites with a typical postfix – dovecot – roundcube setup may handle spam/ham marking when performed via the web interface (roundcube), but mostly fail when performed via an IMAP client like Thunderbird or Outlook.

Unless of course, the spam filter is notified somehow of the user’s action and the message content is fed to the filter’s learning mechanism, which the user thought was happening anyway. This was the problem in this case, and here follows what it took.

A word about the infrastructure. It’s a small site, <1000 users and about 50 domains, though there would be any problem with more users and/or domains. The server is using dovecot and rspamd with the fuzzy_check module enabled and a local fuzzy database, in addition to the configured by default rspamd servers. With small changes the solution can be applied to a bayes module or spamassassin.

The first problem was how to detect when a user drags a message out of the Junk folder and into INBOX, no matter how this happens. Normally this is not logged anywhere, which is a problem, unless a block like the following is added in dovecot’s configuration file (/etc/dovecot/dovecot.conf in this case):

plugin {
    mail_log_events = copy
    mail_log_fields = uid box msgid size
}

After restarting dovecot when a user moves a message to the Trash or, in our case, from Junk to INBOX, a line like the following will be logged:

Nov 12 09:37:23 mailhost dovecot: imap(xxx@nhyui.com)<4708>: copy from Junk: box=INBOX, uid=131400, msgid=<921774d7b58b28e02d4b2bf62.f3e832bb98.20201111125929.6962bf5040.91721499@mail…, size=76041

This is a line that says user xxx@nhyui.com moved a message from the Junk folder to INBOX. The message’s UID is shown as uid=131400. Now what we need to do is scan the log for lines like this, preferably in real time, and act upon them. Here is where the word “abusing” in the title is derived from.

Fail2ban is not designed for this purpose, but chances are it is already installed and running on most internet facing servers and if not, it is available in most, if not all, repositories. And its purpose in life is to follow log files and block offending IP addresses. Since it scans log files, why not trace mailbox actions too? All it needs is a filter to find the ‘dovecot….copy’ lines and a rule to act upon these lines. To get the text of a specific message from dovecot we need to know the owner of the message, the message’s uid and the mailbox where the message resides. Let’s assume she’s dragging a message from Junk to INBOX and go from there (from INBOX to Junk is practically the same with a few minor modifications).

In /etc/fail2ban/filter.d create a file hamfilter.conf and put the following in it:

[INCLUDES]
before = common.conf
[Definition]
_auth_worker = (?:dovecot: )?auth(?:-worker)?
_daemon = (?:dovecot(?:-auth)?|auth)
EMAIL = \w+(\.\w+)?@\w+(-\w+)?\.\w+

failregex = dovecot: imap\(<F-USER><EMAIL></F-USER>\).*: copy from Junk.* box=INBOX, uid=<F-ID>\d+</F-ID>, msgid=.*size=\d+

[Init]
journalmatch = _SYSTEMD_UNIT=dovecot.service

This filter will create 2 variables, F-USER and F-ID. Now in /etc/fail2ban/actions.d create a file named hamaction.conf, and put the following in it:

[Definition]
actionban = doveadm fetch -u <F-USER> text uid <F-ID> mailbox INBOX | rspamc -f 3 -w 15 fuzzy_add
actionunban = /bin/true
[Init]

In this local fuzzy database, 3 is the L_FUZZY_WHITE tag, meaning the good messages.

Lastly, edit /etc/fail2ban/jail.local and add the following lines at the end of the file:

[mark_ham]
enabled = true
maxretry = 1
findtime = 1m
filter = hamfilter
action = hamaction
logpath = /var/log/mail.log

In case you log with systemd’s journal, the logpath should be replaced with the appropriate logtarget name.

[mark_ham]
enabled = true
maxretry = 1
findtime = 1m
filter = hamfilter
backend = systemd
logtarget = dovecot

In this system dovecot logs using the mail facility in /var/log/mail.log, but not all systems may do the same, so adjust accordingly. Now just reload fail2ban and the mark_ham jail will start running. If you drag a message from Junk to the INBOX, rspamd will learn about it almost instantly.

This method can also be tuned to train a Bayes filter, with rspamd keeping each user’s statistics separately.

pfSense hard limit

The problem: data caps. Data traffic over mobile networks comes at a cost. So, when X MB or GB of data have been downloaded in a time period (day or week), all “client” traffic stops. Traffic shapers don’t help in this case, and vnstat is not versatile enough. pfSense has to be reachable from the outside for admin purposes, but the users will have no more access to the internet. When the time period expires pfSense resumes normal operation. It should also be reboot resilient, since reboots zero out all internal counters. Interestingly, instead of the usual culprits, python and php, awk came to the rescue. Who would have guessed. So, after some tinkering and reading, the following script came to existence. The function gecko() below is not necessary, but I wanted to try an awk function.

#!/usr/bin/awk -f

function gecko(filename)
{
    printf("0\n0\n0\n0\n") > filename
}

BEGIN {
    # First positional argument -> file where traffic data is stored
    # Second positional argument -> traffic limit (in MB)
    # Third positional argument -> base period (defaults to weekly)
    # line 1 (date0) -> last time we operated on this file
    # line 2 (date1) -> current time of operation on this file
    # line 3 (old_data) -> bytes moved before last counter reset
    # line 4 (data) -> data moved (new measurement)
    statfile = ARGV[1]
    limit = ARGV[2]
    period = ARGV[3]
    daily = 86400
    weekly = 604800
 
    if(period == "daily"){
        period=daily
    } else {
        period=weekly
    }
 
    getline date0 < statfile
    getline date1 < statfile
    getline old_data < statfile
    getline data < statfile

    # Epoch time: Thu 197001010000
    "date +%s" | getline t_epoch

    # In this case wa want a weekly reset every Saturday at 00:00, hence 86400*2=172800
    t_since_zero = (t_epoch-172800) % period
    if(date0 > t_since_zero || date0 == "" || date1 < date0){
        gecko(statfile)
        exit
    } else {
        date0 = date1
        date1 = t_since_zero
        # ***** WARNING *****
        # The interface monitored in this configuration is `ue0`
        # Change manually according to current configuration
        while(("pfctl -s Interfaces -i ue0 -vv" | getline data_new) > 0){
        if(data_new ~ /In4\/Pass/){
            gsub(/^.*Bytes: /, "", data_new)
            gsub(/ +\]$/, "", data_new)
            data_new = data_new / 1048576
            if(data_new < data){
                old_data = old_data + data
            }
            data = data_new
        }
    }

    if(data + old_data > limit){system("/sbin/pfctl -d")} else {system("/sbin/pfctl -e")};
        printf("%d\n%d\n%d\n%d\n", date0, date1, old_data, data) > statfile;
    }
}

Put the above in a file, i.e. /root/caps.awk, and

chmod 755 /root/caps.awk

Then put in crontab a line like this:

*/2 * * * * root /root/caps.awk /root/vol.txt 2000

which means “turn off pf if traffic exceeds 2000MB.