You were never here, you didn't see anything..

pfsense & ELK


I recently came across a blog post that inspired me to install ELK on a server and pipe the log data to it from pfsense. Fantastic article, fantastic dashboard and wonderful inspiration. I banged my way through things armed with the blog post and several Internet searches. In the end I now have this to admire at will:

ELK and pfsense
ELK and pfsense

I have made several modifications along the way as I plan to break out and visualize much more than just the firewall data which is seen in the above screen shot. More to come on that later hopefully!

Some reference pages:

And the author of the original post that inspired me to get moving is working on updates:

Update 1:

I changed the map display to be heat-map from “Scaled Circle Markers”. Looks much nicer:

pfsense & ELK #2
pfsense & ELK #2

Update 2:

Tracking other events by time:

Service Events by Time
Service Events by Time

Update 3:

Grok patterns file can be found HERE

Logstash configuration files:

There is a section at the bottom of the patterns file for radius log entries. I have it commented out as pfsense is no longer acting as my radius server. The more recent versions of freeradius properly support TLS and those are not available on pfsense as of yet. The largest issue this causes is android v6.x clients can not authenticate.

Update 4:

I had to rebuild my ELK server (unexplained death). After some reading I have exported the external firewall dashboard. The file can be found : here

Update 5:

  • Current Dashboard – here
  • Visualizations Export – here

Update 6:

  • Saved Searches Export – here

Have any Question or Comment?

13 comments on “pfsense & ELK

Abert

awesome i really did like your updates did you modify the grok patterns?

Reply

I did modify the grok patterns file used in the original articles I followed. I added a lot for parsing out freeRadius and a few other log entries that were giving me useless output.

I will see about getting the patterns file posted soon…

Reply

Any changes for posting the dashboard?

I have all the ELK installed and my pfSense is sending logs but I’m really bad at building the dashboard right now.

Reply

@aredan: That should be doable. I will admit however I have never done that before so I need to figure out how. Also, things have changed a little as I have had to rebuild the ELK machine and have moved my radius services to another machine.

The share URL it gives me is insanely large and the server is not externally accessible.

Reply
tomas

Could you export your visualization config as well? Tried to import your dashboard, but it doesn’t work without the visualization config

Reply

tomas – Post has been updated with the current export for both the Dashboard and the visualizations. The visualizations are all in a single export as I am a bit too lazy to do them individually plus I figured it would make it easier to import them as well.

Reply
Albert Yang

Hi,
Wonderful stuff But I tried adding your files and shows no information on kibana but when i run tails i get the logs showing but no visual?

Thank you

http://postimg.org/gallery/5d6ay18/

Reply

HI,

is that working with the actual pfsense release, ES 2.3 Kibana 4.5 and Logstash 2.3? Because i run a serval problems

Reply
João

Hi,

I have installed pfsense 2.3 and kibana 4.5.
Do you have any update for the grok patterns file for this version of pfsense?

Thanks a lot for your time!

Reply

I have made no updates since the files above.

As for the exported files shared, if there is a guide that I can use to make sure I pull the proper files that would be great. My guess would be the visualizations are broken somehow or maybe the searches are not right. Post updated with my saved searches.

Version in use at present are:
Kibana 4.4.2
Logstash 2.2.4
Elasticsearch 2.3.4
PFSense 2.3.2

Reply
Daniel

Thanks for these files! They really helped to get my ELK up and running. I had two questions about them. First the files reference geoip.country_code3.raw, but that field isn’t being indexed. I do see geoip.country_code3. Are we supposed to manually add that field? Second, are your EM7 and EM11 both external interfaces?

Reply
Andy

Hi and thank you so much for this! Until I found this site, I had looked at so many articles on how to install ELK and configure with pfSense and have spent many hours trying to put the missing pieces in place only to fail. With the information you provided, I had an ELK stack running in about 5 mins!

I have one question, I cannot seem to get the GeoIP information displayed on the dashboard. I am seeing this information when I perform a search but it doesn’t show on the dashboard. I am using the dashboard that you posted.

Can you think of anything that I am missing?

Thanks,
Andy

Reply

Daniel – If the raw fields don’t show up if I recall you have to refresh the index. this can be done via the settings > indices page. There is a button for it at the top.

As for external interfaces, I am only using em11. em7 is an interface I move around for testing and such.

Andy – They changed the way the map API is handled. Mine quite working for a while and I had to go in and change the map settings to use another configuration. I don’t recall the details off hand, but the hunt started with the following link:

https://github.com/StamusNetworks/SELKS/wiki/Kibana-4.5—GeoIP-Map-visualizations-not-working

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This blog is kept spam free by WP-SpamFree.

Most who fail…

Most who fail have yet to really try!