Sending pfSense logs to the DShield project
02June2017 – Originally posted
28Nov2017 – Updated due to script changes
What is DShield and why would I send them my logs?
According to the SANS Internet Storm Center (ISC), “DShield provides a platform for users of firewalls to share intrusion information. DShield is a free and open service.” While DShield is often referred to generically as a “collaborative firewall log correlation system,” for all practical purposes, it is a bit of threat intelligence well before threat intelligence was cool or an overused buzzword. 😉 The truth is many folks across the internet provide data to the ISC and it really is a community-based effort. DShield aggregates/parses the data and determines different attack vectors and trends. This information then gets filtered back down to the community and it is often found in many of the threat/IP blacklist feeds used by security products. How cool is that? So the bottom line is that if you are a consumer of this data, why not help out the cause and contribute to it? You also get the added bonus of seeing firewall reports on your own last 30 days of data via your account portal. I’ve included one example of the many reports you might expect at the bottom as well. Did I mention it’s free?!?!
DShield account creation
Get started by creating an account on DShield. Once again, it’s free.
After your account is created, go to ‘my account’ and you will see the information we need. Namely, the API key and the user ID. I would recommend just leaving this page up and copying/pasting these when we need them later.
pfSense web config
To get started, we’ll need to add in a notification email address on pfSense. You can do this by going to Systems->Advanced->Notifications. If you already have an account configured here, great! Just remember you will need to use that same email address later when configuring the script. If you don’t have anything configured here, you will need to add one. I chose to use a Gmail address and if you want to do the same, make the modifications as shown in the picture below. Don’t forget to send a test after saving your contents!
Note: I used a different Gmail address than my standard use one for the simple fact you can’t use an account that has 2-factor authentication enabled.
SSH on pfSense is not enabled by default. You can easily turn it on via the System->Advanced menu.
Command line config
Now, SSH to your pfSense box using PuTTY or another SSH client. Don’t forget your username is root, not admin.
Once you are logged in, go to option 8, which drops you to a standard command line. Create a bin directory under root and then change to that directory
# mkdir bin # cd bin
Next, download the config file from GitHub with the first command and make the file executable with the second command.
# curl https://raw.githubusercontent.com/jullrich/dshieldpfsense/master/dshield.php > /root/bin/dshieldpfsense.php chmod +x dshieldpfsense.php
Next, download the sample ini file from GitHub.
# curl https://raw.githubusercontent.com/jullrich/dshieldpfsense/master/dshield.sample > /root/bin/dshield.ini
With the latest version of the script, there is no need to edit the actual php script. Instead, you edit the ini by adding your apikey (API key from portal), uid (User ID from portal), and fromaddr (from email address in notifications). As stated earlier, it is far easier to copy/paste the apikey and uid values from the DShield page.
Note: double-check the interface if you are using anything other than WAN. You can use multiple interfaces as long as you delimit them with commas.
# vi dshield.ini
Once those lines are changed, go ahead and exit vi. Perform a quick test run using the command below.
If everything is configured correctly, you can take a look at the system log. The command below will grep back any DShield-related content.
# clog /var/log/system.log | grep dshield
The only issue is that if it worked, you will not see anything in the log. 🙁 So go ahead and run the same command again. Assuming no new activity occurs in that time, your system log will now have a log entry similar to the one below the command.
# clog /var/log/system.log | grep dshield pf php: ./dshieldpfsense.php: no new lines added to log since last run OK
If that didn’t work, there are a few other items you can try.
First, you can modify the dshield.ini file and add a ccaddr so the reports go to that email as well. This is extremely useful when you feel the data is collected and correct, but nothing is showing on the DShield website after a fair amount of time.
Another great way to troubleshoot is by changing “debug=0” to “debug=1” which is also found in the dshield.ini file. After making the change, re-run the dshieldpfsense.php script. This time around you should see a fair amount of information returned similar to below. If you see why no logs were sent, e.g. only LAN or ICMP traffic was seen recently, then change debug back to 0.
Reading Nov 29 23:00:54 pf filterlog: 5,,,1000000103,igx0,match,block,in,4,0x0,,64,22100,0,DF,6,tcp,52,192.168.1.2,126.96.36.199,56363,443,0,FA,2164609854,1629822578,1369,,nop;nop;TS Log was rejected due to wrong interface or flags or because it is ICMP: protocol TCP interface LAN flags FA Reading Nov 29 23:00:54 pf filterlog: 5,,,1000000103,igx0,match,block,in,4,0x0,,64,3647,0,DF,6,tcp,52,192.168.1.2,188.8.131.52,56361,443,0,FA,3134666011,2501649167,1550,,nop;nop;TS Log was rejected due to wrong interface or flags or because it is ICMP: protocol TCP interface LAN flags FA Reading Nov 29 23:00:54 pf filterlog: 5,,,1000000103,igx0,match,block,in,4,0x0,,64,25539,0,DF,6,tcp,52,192.168.1.2,184.108.40.206,56362,443,0,FA,4017349769,959963474,1550,,nop;nop;TS Log was rejected due to wrong interface or flags or because it is ICMP: protocol TCP interface LAN flags FA Reading Nov 29 23:00:55 pf filterlog: 5,,,1000000103,igx0,match,block,in,4,0x0,,64,22101,0,DF,6,tcp,52,192.168.1.2,220.127.116.11,56363,443,0,FA,2164609854,1629822578,1369,,nop;nop;TS
A few other file locations worth mentioning for troubleshooting…
/tmp/lastdshieldlog <- contains the last log sent
/var/run/dshieldlastts <- contains the last time (in epoch) the script was completed
Automate it with cron
If everything appears to be working, you just need to automate it. Do this by typing in ‘crontab -e’ from the command line and copying in the cronjob below. I would recommend making these values (25,55) somewhat random and leave them 30 minutes apart from one another. For example, 4,34 or 12,42 or 27,57 would work great.
25,55 * * * * /root/bin/dshieldpfsense.php
Exit as you would with vi and that should give you a message of ‘crontab: installing new crontab’
And now you wait… After your cronjob runs a few times and enough data is collected, you can take a look at your data as aggregated by DShield.
If you or any feeds you use consume data from DShield, contributing to the cause is really the right thing to do! Keep in mind that other firewalls can send their data as well and it is not just limited to pfSense. For a complete rundown on additional clients and other in’s and out’s of the DShield project, visit the link below.
Dallas Haselhorst has worked as an IT and information security consultant for over 20 years. During that time, he has owned his own businesses and worked with companies in numerous industries. Dallas holds several industry certifications and when not working or tinkering in tech, he may be found attempting to mold his daughters into card carrying nerds and organizing BSidesKC.
10 thoughts on “Sending pfSense logs to the DShield project”
SANS Internet Store Center? Sounds a bit off…
They “store” logs, right? LOL Thanks for bringing it to my attention!
Im getting following error in pfsense 2.3.5 rel 2
/root/bin/dshieldpfsense.php: Could not send DShield logs to firstname.lastname@example.org — Error: 553 5.1.2 The recipient address is not a valid RFC-5321 address. d123-v6sm2974057wmd.9 – gsmtp
Aug 10 21:25:03 FW php: /root/bin/dshieldpfsense.php: no new lines added to log since last run OK
The error you are receiving is part of an if/else at the bottom of the PHP script. The fact that you still receive the successful output of “no new lines” is odd. The script changed a while back because of 2.4 and IIRC, it was specifically related to some modifications to the SMTP variables. Are you able to upgrade to 2.4 or later to see if that takes care of it?
Since i have 32bit processor, i have no room to upgrade to 2.4 or later.
Any further thoughts on how this can be resolved or is there any other alternative community based cloud threat intelligence to feed my Pfsense logs.
It appears there are some older versions of the DShield script still out on the net (not just below). This one in particular is from around April 2016 and should be for versions around 2.3 and later. Just a heads up that in older script versions, the authkey, fromaddr, and uid variables were in the script itself and *not* the dshield.ini. Everything else should work the same though, i.e. newer versions of the script changed due to pfSense and not the DShield API.
Should the txt file be changed to php. I have made several attempts but getting the same error as mentioned in earlier post.
Here is the output so far
Exception: Zend Extension ./dshieldpfsense.php does not exist
[2.3.5-RELEASE][email@example.com]/root/bin: clog /var/log/system.log | grep dshield
Aug 21 16:25:19 fw php: /root/bin/dshieldpfsense.php: Could not send DShield logs to firstname.lastname@example.org — Error: 553 5.1.2 The recipient address is not a valid RFC-5321 address. 1-v6sm5469718wmf.47 – gsmtp
I’m assuming the previous discussion regarding the script on version 2.3 was you as well. Yes, the script should be php instead of txt. If you haven’t already, you will want to chmod 755 the file as well. The exception error may require you to add the interpreter to run properly, e.g. ‘php ./dshieldpfsense.php’
Is this still in support?
Hey John! I haven’t tried it for a long time. Let me know if it’s no longer working and I’ll update the post as such. Thanks!