syslog-ng is an open source syslog server that can be installed on many *nix servers. In my example, I installed it on a 64bit CentOS 6.3 server with minimal package.
1) Download and extract the source code.
tar xvzf eventlog-0.2.12.tar.gz
tar xvzf syslog-ng-3.4.0alpha3.tar.gz
2) Install dependencies.
yum -y install kernel-headers* glibc-headers-* glibc-devel-* gcc-* zlib-* zlib-devel-* libffi-* libffi-devel-* e2fsprogs-devel-* keyutils-libs-devel-* libselinux-devel-* libsepol-devel-* libselinux-devel-* krb5-devel-* openssl-devel-* pcre-devel xz
wget ftp://ftp.gtk.org/pub/glib/2.32/glib-2.32.1.tar.xz
tar xvJf glib-2.32.1.tar.xz
cd glib-2.32.1
./configure
make
make install
3) Install eventlog.
cd eventlog-0.2.12
./configure
make
make install
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
4) Install syslog-ng.
cd syslog-ng-3.4.0alpha3
./configure
make
make install
5) Create the syslog-ng configuration file.
cd /usr/local/etc
mv syslog-ng.conf syslog-ng.conf.bak
mkdir /var/log/syslog-ng
vi syslog-ng.conf
@version:3.4
@include “scl.conf”
####################################
##Source
source remote {
udp(ip(0.0.0.0) port(514));
tcp(ip(0.0.0.0) port(514));
};
####################################
##Destination
destination host01 {
file(“/var/log/syslog-ng/host-01.log”);
};
destination host02 {
file(“/var/log/syslog-ng/host-02.log”);
};
destination host03 {
file(“/var/log/syslog-ng/host-03.log”);
};
####################################
##Filter
filter host01 {
host(“host-01_server_name“);
};
filter host02 {
host(“host-02_server_name“);
};
filter vmh03 {
host(“host-03_server_name“);
};
####################################
##Log
log {
source(remote);
filter(host01);
destination(host01);
};
log {
source(remote);
filter(host02);
destination(host02);
};
log {
source(remote);
filter(host03);
destination(host03);
};
6) Check the /var/log/syslog-ng folder. The remote host’s syslog files should be stored there now.
7) The open source edition syslog-ng does not come with a web UI, so you have to install it separately. I used a simple tool named logstash.
wget http://semicomplete.com/files/logstash/logstash-1.1.1-monolithic.jar
8) Create the logstash configuration file (logstash-syslog.conf) in the same folder as where the jar file is.
input {
file {
type => “linux-syslog”
path => [ “/var/log/syslog-ng/*.log” ]
}
}
output {
stdout { }
elasticsearch { embedded => true }
}
9) Run logstash.
java -jar logstash-1.1.1-monolithic.jar agent -f logstash-syslog.conf — web –backend elasticsearch:///?local
10) Go to http://<server_IP>:9292, and use the Lucene’s string query language to search.
Jackie Chen's IT Workshop 
Your yum install needs to include xz. Can’t unzip glib without it.
Not really. ‘ tar xvJf ‘ can do that.
Thank you for pointing this out. Yes, you need to install xz first. It was installed on my server, but I did not realize it.
Apparently no it can’t….
Machine#tar xvJf glib-2.32.1.tar.xz
tar (child): xz: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
-rw-r–r– 1 root root 6138200 Oct 8 21:05 glib-2.32.1.tar.xz
The file is there but I can’t get it to engage and break it out.
I think Tom is right, you need to install xz first by running ‘yum -y install xz’, ‘tar xvJf’ actually calls xz when extracting the .xz package.
That fixed it. Thanks.
Really nice write up. I could see that my syslog server is receiving logs through tcpdump but not in /var/log/syslog-ng folder. I even went and created a file that correspond to the /var/log/syslog-ng/host-02.log but still does not write to a any log files. I did some research and see that others have their syslog-ng.conf in /etc/syslog-ng/ folder; whereas yours is in /usr/local/etc. I assume this doesn’t affect the install. I also tried to restarting the rsyslog service but still no content in my log file. Below is out my sysconfig file. Any suggestion is greatly appreciated. Thanks!
@version:3.4
@include “scl.conf”
####################################
##Source
source remote {
udp(ip(0.0.0.0) port(514));
tcp(ip(0.0.0.0) port(514));
};
####################################
##Destination
destination pacore1 {
file(“/var/log/syslog-ng/core.log”);
};
####################################
##Filter
filter pacore1 {
host(“192.168.1.1“);
};
####################################
##Log
log {
source(192.168.1.1);
filter(core);
destination(core);
};
It should not work as you used the wrong destination name and filter name in your log sections. The correct one should be:
##Log
log {
source(remote);
filter(pacore1);
destination(pacore1);
};
Awesome! Thanks Jackie.
I am having problems with this command: java -jar logstash-1.1.1-monolithic.jar agent -f logstash-syslog.conf — web –backend elasticsearch://10.1.15.80/?local
I get this:
“LogStash::Runner” org.jruby.exceptions.RaiseException: (InvalidOption) invalid option: -backend
If I run: java -jar logstash-1.1.1-monolithic.jar agent -f logstash-syslog.conf
It runs fine but netstat shows nothing listening on 9292
it should be ‘–backend’ (double dash), the page does not show it correctly.
Thanks Jackie!
Sorry Jackie, help again…
What is between logstash-syslog.conf & web? It shows as “logstash-syslog.conf — web” but I am trying various combos to no avail.
Thanks.
It is double dash as well. Good luck 🙂
Hi Jackie,
when i run the logstash i get the below error.
Exception in thread “LogStash::Runner” org.jruby.exceptions.RaiseException: (InvalidOption) invalid option: –web
It is double dash before web. Somehow this wordpress theme shows double dash as single dash.
I did put double dashes in front of both web and backend but i still got the error, below is the full output:
Exception in thread “LogStash::Runner” org.jruby.exceptions.RaiseException: (Inv alidOption) invalid option: –web
at OptionParser.complete(file:/home/kunle/logstash-1.1.1-monolithic.jar! /META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1459)
at org.jruby.RubyKernel.catch(org/jruby/RubyKernel.java:1197)
at OptionParser.complete(file:/home/kunle/logstash-1.1.1-monolithic.jar! /META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1457)
at OptionParser.parse_in_order(file:/home/kunle/logstash-1.1.1-monolithi c.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1271)
at org.jruby.RubyKernel.catch(org/jruby/RubyKernel.java:1197)
at OptionParser.parse_in_order(file:/home/kunle/logstash-1.1.1-monolithi c.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1264)
at OptionParser.order!(file:/home/kunle/logstash-1.1.1-monolithic.jar!/M ETA-INF/jruby.home/lib/ruby/1.9/optparse.rb:1258)
at OptionParser.permute!(file:/home/kunle/logstash-1.1.1-monolithic.jar! /META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1349)
at OptionParser.parse!(file:/home/kunle/logstash-1.1.1-monolithic.jar!/M ETA-INF/jruby.home/lib/ruby/1.9/optparse.rb:1370)
at OptionParser.parse(file:/home/kunle/logstash-1.1.1-monolithic.jar!/ME TA-INF/jruby.home/lib/ruby/1.9/optparse.rb:1360)
at Agent.parse_options(file:/home/kunle/logstash-1.1.1-monolithic.jar!/l ogstash/agent.rb:192)
at Agent.run(file:/home/kunle/logstash-1.1.1-monolithic.jar!/logstash/ag ent.rb:303)
at logstash.runner.run(logstash/runner.rb:70)
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:258)
at logstash.runner.run(logstash/runner.rb:95)
at logstash.runner.main(logstash/runner.rb:37)
at logstash.runner.(root)(logstash/runner.rb:120)
I don’t know why it does not work for you, it looks to me still has something to do with the double dashes thing. Did you just copy and paste the command line? If so try to type them in.
it’s a double dash web. Make sure there is a space between the dashes, and the word web.
it’s a double dash with a space between the dashes and web.
**A little new to this**
After I wget both syslog-ng and eventlog, I try to extract the files and I get the error message: tar (child): syslog-ng-3.4.0alpha3.tar.gz: cannot open: no such file or directory. I get the same error message for eventlog. What am I not doing correctly?
Where did you store the syslog-ng tar file? Make sure you are in the same folder when extracting the tar.gz file, or use the full path.
I managed to install syslog-ng 3.2 on CentOS. What I need is to install new version with GeoIP support enabled. I am new to Linux and would appreciate if you could let me if this version has GeoIP support, and if not how can I add to it.
I did not use GeoIP with syslog-ng. As far as I know, you need to include ‘–enable-geoip’ when compiling. FYI: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/syslog-ng-ose-v3.4-guide-admin-en/html/reference-template-functions.html
Thanks. When I try to install the dependencies I get the following:
yum -y install kernel-headers* glibc-headers-* glibc-devel-* gcc-* zlib-* zlib-devel-* libffi-* libffi-devel-* e2fsprogs-devel-* keyutils-libs-devel-* libselinux-devel-* libsepol-devel-* libselinux-devel-* krb5-devel-* openssl-devel-* pcre-devel xz
Loaded plugins: fastestmirror, protect-packages, refresh-packagekit, security
Loading mirror speeds from cached hostfile
* atomic: www6.atomicorp.com
* epel: mirror.steadfast.net
* rpmforge: mirror.us.leaseweb.net
Setting up Install Process
No package glibc-headers-* available.
No package glibc-devel-* available.
No package gcc-* available.
No package zlib-devel-* available.
No package libffi-devel-* available.
No package e2fsprogs-devel-* available.
No package keyutils-libs-devel-* available.
No package libselinux-devel-* available.
No package libsepol-devel-* available.
No package libselinux-devel-* available.
No package krb5-devel-* available.
No package openssl-devel-* available.
No package pcre-devel available.
Nothing to do
What I am doing wrong.
You did not do anything wrong. What it means is that those packages are not available in your yum repositories.
Go to ‘/etc/yum.repos.d/’, and check what are there. The default CentOS yum repositories are: CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Media.repo CentOS-Vault.repo
Hi
Following is what I have in my /etc/yum.repos.d/. How to do I add the other repositories? Thanks in advance for your guidance
-rw-r–r– 1 root root 119 Jul 27 2011 afni-local-rhel-base.repo
-rw-r–r– 1 root root 957 Nov 4 21:52 epel.repo
-rw-r–r– 1 root root 1056 Nov 4 21:52 epel-testing.repo
It seems someone has changed the yum repository in your server. If you want to add more repo sources, firstly find your server version then either copy the repo files from other server running on the same version or download and install the repo rpm packages .
Thanks. I got the repos. I have another newbee question. To install GeoIP support, I will change the step four of your instructions as follows – right?
4) Install syslog-ng.
cd syslog-ng-3.4.0alpha3
./configure
make
make install –include enable-geoip
I think it should be ‘./configure –enable-geoip’ (double dashes in front of enable).
I’m using syslog-ng-3.3.8 and eventlog-0.2.12. I’ve installed all of the dependencies and when I do ./configure –enable-dynamic-linking for syslog-ng I get the following error at the end of the run
configure: error: Package requirements (eventlog >= 0.2.12) were not met: No package ‘eventlog’ found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables EVTLOG_CFLAGS and EVTLOG_LIBS to avoid the need to call pkg-config
Any help here would be appreciated.
I guess you missed this command ‘export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH’ in step 3).
I’m pretty sure i did that.. Anyway to check?
Before you compile the syslog-ng, check the PKG_CONFIG_PATH – echo $PKG_CONFIG_PATH
I am having some trouble getting the modules/afsocket to compile. Getting the following error:
In file included from /usr/include/net/ethernet.h:27,
from /usr/include/libnet.h:97,
from afinet-dest.c:43:
/usr/include/linux/if_ether.h:110: error: expected specifier-qualifier-list before ‘__be16′
make[4]: *** [libafsocket_notls_la-afinet-dest.lo] Error 1
make[4]: Leaving directory `/usr/local/src/mongo/syslog-ng-3.4.1/modules/afsocket’
make[3]: *** [all] Error 2
make[3]: Leaving directory `/usr/local/src/mongo/syslog-ng-3.4.1/modules/afsocket’
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/usr/local/src/mongo/syslog-ng-3.4.1/modules’
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/local/src/mongo/syslog-ng-3.4.1′
make: *** [all] Error 2
This occurs when it is compiling the afinet-dest.c in modules/afsocket.
Any ideas ?
I am trying to get multiple hosts to log into a single file as they are part of a cluster and any one of the host should be able to send a syslog message to this server. Is what I have here viable? I didn’t seem to work?
@version:3.4
@include “scl.conf”
####################################
##Source
source remote {
udp(ip(0.0.0.0) port(514));
tcp(ip(0.0.0.0) port(514));
};
####################################
##Destination
destination test{
file(“/var/log/syslog-ng/test);
};
####################################
##Filter
filter host01 {
host(“10.244.146.161“);
};
filter host02 {
host(“10.244.146.162“);
};
filter host03 {
host(“10.244.146.163“);
};
####################################
##Log
log {
source(remote);
filter(host01,host2,host3);
destination(test);
};
TIA
hi Jackie,
following up with your config, I got smoothly in the installation but then I checked under /var/log/syslog-ng, NO file was stored there.
Next I did “netstats -na | grep 514” and see that the port is not up yet. Do we need to reboot the machine after installed?
I could see the message came from my remote host but somehow it got unreachable on this server.
05:43:24.086849 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 167) 2a01:d0:890e:8000::4 > 2a01:d0:890e:20::5: [icmp6 sum ok] ICMP6, destination unreachable, length 167, unreachable port, 2a01:d0:890e:8000::4 udp port 514
05:43:24.086869 IP6 (hlim 63, next-header UDP (17) payload length: 98) 2a01:d0:890e:20::5.514 > 2a01:d0:890e:8000::4.514: [udp sum ok] SYSLOG, length: 90
Sounds to me the syslog-ng is not running in your server. Try to manually start the process: /usr/local/syslog-ng/sbin/syslog-ng, and see how it goes.
Hi, i have problems with local loogin.. I´m trying with
source local { unix-stream(“/dev/log”); internal(); udp();};
destination host01 {file(“/var/log/auth.log”);};
filter f_auth { facility(auth, authpriv);};
log {source(local);filter(f_auth);destination(host01);};
I want to loggin local authentication only. When i start the daemon it never create the auth.log file..
Help me please..
Thanks
Does it work to CentOS 5.9 release too?
Not sure, check their website.
Here again, i’m having trouble with the instruction ‘make’ to syslog..alfa. The following message appears: “make *** : no targets specified and no makefile found. Stop”
Did you forget to run ./configure to compile?
No, actually I had to update yum with “yum update yum” command, and after that ‘yum update’ to update all dependecies, then I started the process all over again and I got it right. Thats a good tip for those who can’t run some command. But I’m having another trouble here.
@version:3.4
@include “scl.conf”
####################################
##Source
source remote {
udp(ip(0.0.0.0) port(514));
tcp(ip(0.0.0.0) port(514));
};
####################################
##Destination
destination nocserver {
file(“/var/log/syslog-ng/nocserver.log”);
};
####################################
##Filter
filter nocserver {
host(“nocserver.trt20.intra”); // in here does it have to be my ip address or not?
};
####################################
##Log
log {
source(remote); // in here does it have to be “remote” or my ip address?
filter(nocserver);
destination(nocserver);
};
When I configure my syslog-ng.conf, my log file does not appear. Is there something wrong?
Thanks for helping anyway
Any writeups to use syslog-ng with MySQL + loganalyzer ?
It would be really helpful.
Regards.
Hi Jackie….
I am getting following error while compiling syslog-ng-3.4.0alpha3. I am a rookie… please help me out
checking for LIBDBI… no
checking for dbi_initialize in -ldbi… no
checking whether to enable SQL support… no
checking for GLIB… no
configure: error: Package requirements (glib-2.0 >= 2.10.1 gmodule-2.0 gthread-2.0) were not met:
No package ‘glib-2.0’ found
No package ‘gmodule-2.0’ found
No package ‘gthread-2.0’ found
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.
Alternatively, you may set the environment variables GLIB_CFLAGS
and GLIB_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
[root@sf-log-nms syslog-ng-3.4.0alpha3]#
I have some problems running the comand ” java -jar logstash-1.1.1-monolithic.jar agent -f logstash-syslog.conf — web –backend elasticsearch:///?local ” have this ans [root@localhost 29112013]# java -jar logstash-1.1.1-monolithic.jar agent -f logstash-syslog.conf —web –backend elasticsearch:///?local
Aborted (core dumped)
pls need some help
Hi, I followed your instruction to install and configure syslog-ng. My question is how to verify the installation is succeeded and running? Because no log is created in /var/log/syslog-ng.
Thank you.
i have the core dump error. Please reply need help.
Thanks alot ,good work… but I think this work not enough to make syslog-ng running on linux … if you follow the step as you mention above you not have the service who run syslog-ng in /etc/init.d/ that mean you just collect log files and show it in logstash ….. in addetion the service of logsys-ng sure unuseable in this case …
Regards,
Hi Nikhil , Core dump error means you need to update your java .. if you check your java version it will be 1.5 or less update it to 1.7 and make sure that you have : OpenJDK Runtime Environment OpenJDK 64-Bit Server .. its depend on your OS ..
just update your java then everythings will be fine to you ….. 🙂
Good Luck