We use ElasticSearch at my job for web front-end
searches. Performance is critical, and for our purposes, the data is mostly
static. We update the search indexes daily, but have no problems running on
old indexes for weeks. The majority of the traffic to this cluster is
search; it is a "read heavy" cluster. We had some performance hiccups at
the beginning, but we worked closely with Shay Bannon of ElasticSearch to
eliminate those problems. Now our front end clusters are very reliable,
resilient, and fast.
I am now working to implement a centralized logging infrastructure that
meets compliance requirements, but is also useful. The goal of the logging
infrastructure is to emulate as much of the Splunk functionality as
possible. My previous write-up on
explains why we decided against Splunk.
After evaluating a number of options, I've decided to utilize ElasticSearch
as the storage back-end for that system. This type of cluster is very
different from the cluster we've implemented for heavy search loads.
If you haven't looked at OSSEC HIDS, here's the overview:
OSSEC is a scalable, multi-platform, open source Host-based Intrusion
Detection System (HIDS). It has a powerful correlation and analysis engine,
integrating log analysis, file integrity checking, Windows registry
monitoring, centralized policy enforcement, rootkit detection, real-time
alerting and active response.
It runs on most operating systems, including Linux, OpenBSD, FreeBSD, MacOS,
Solaris and Windows.
OSSEC is a great product, but I ran into an issue when attempting to fulfill
a require for PCI-DSS which involved reviewing our LDAP logs. I knew
OSSEC would make this simple. I started writing a rule and realized I had
hit a significant roadblock. OpenLDAP logs events as they happen and only
logs data relevant to that particular event. A connect event has the ports
and IPs, and the bind event contains the username, but only the connection
id is the same in the two events.
I do most of my work over SSH. Even when I'm working in my browser or
pgAdminIII, I'm usually doing that over SSH tunnels. VPN Software has been
around for quite some time and it's still mostly disappointing and usually run
by the least competent group in any IT department. I developed a workflow using
SSH from my laptop, either on the corporate network or at home, I can ssh
/directly/ to the server I'm interested in working on.
In order to accomplish this, I have made some compromises. First off, if I'm
SSH-ing from my home, I am /required/ to type the fully qualified domain names
(FQDN) when workign remotely. I use the presence of the domain name to activate
the proper leap frogging. I also decided to use ControlMaster's with SSH that
can leave me with a terminal without a prompt when I forget which shell is my
master. Overall, the pros outweigh the cons and I'm more productive because of
Using a ControlMaster with ssh allows multiple connections to the same tcp
connection. This means subsequent connections are much faster to open, but
places a limit on the original connection that all connections riding on it must
be closed before the ControlMaster connection closes. This may or may not be
desirable, but does come in handy when using ProxyCommand to bounce around
through jump hosts as the connection establishment overhead is removed.
First things first. I've stated that you should drop everything and install Graphite. If you didn't already, please do that now. Go ahead, I'll wait.
Good? Good. I don't frequently insist on anything like I do with Graphite. There's a lot of reasons for that. If you don't believe me, please see @obfuscurity's awesome Graphite series on his blog.
When you get back we'll talk about how to monitor ElasticSearch with Graphite for fun and profit!
The reaction to my Central Logging
post has been significantly greater and more positive than I could've expected,
so I wanted to recap some of the conversation that came out of this. I am
pleasantly surprised by most of the comments on the Hacker News Thread. So, here's a real quick
recap of the responses I've received. I will continue this series this weekend
with more technical details.
I have worn many hats over the past few years: System Administrator, PostgreSQL and MySQL DBA, Perl Programmer, PHP Programmer, Network Administrator, and Security Engineer/Officer. The common thread is having the data I need available, searchable, and visible.
So what data am I talking about? Honestly, everything. System logs, application logs, events, system performance data, and network traffic data are key requirements to making any tough infrastructure decision, if not key to the trivial infrastructure and implementation decisions we have to make everyday.
I'm in the midst of implementing a comprehensive solution, and this post is a brain dump and road map for how I went about it, and why.
It's true isn't it? In the name of serving pages faster, strip out unnecessary
CPU cycles from rendering the underlying HTML. How often do you actually post
to your blog anyways? That's what I thought.
This post is brought to you by HiD which is a
Perl version of the Jekyll Dynamic to Static content management system.
I've been contemplating moving my server to a new provider for a very long
time. It's a lot of work and I wasn't sure I wanted to deal with it. After
a week of prep work, I finally made the move to
Linode.com. There are a number of reasons for this
I married a Statistician, so this
article sums the
lectures I receive on a daily basis. Risk Management is statistical analysis,
and I'm not sure how many folks in IT Security have Graduate level Stat
exposure. So, the understanding of our statistical shortcomings is key. You
need to read that entire article, twice. This statement struck me, as I've
noticed a scary trend in IT Security:
"People who know a little bit of statistics - enough to use statistical
techniques, not enough to understand why or how they work - often end up
horribly misusing them. Statistical tests are complicated mathematical
techniques, and to work, they tend to make numerous assumptions. The
problem is that if those assumptions are not valid, most statistical tests
do not cleanly fail and produce obviously false results."
As we outsource more security, and buy more products, we must be careful, as this statement is also true:
"People who know a little bit of IT Security - enough to use an IDS or
SIEM, not enough to understand why or how they work - often end up
horribly misusing them. Security tools use complicated technical
techniques, and to work, they tend to make numerous assumptions. The
problem is that if those assumptions are not valid, most security tools do
not cleanly fail and produce obviously false results."
My wife's constant guidance in Statistics has been invaluable to my evaluations
of IT Security Policy and Implementation. When I came across this article
thanks to @alexhutton, I had to share it!
For several years I've managed to bend cfEngine
2.0's architecture to my will. Being an experienced Perl programmer, I was
able to abuse the configuration language snytax in order to accomplish a number
of strange things including copy back and automated management of
OSSEC-HIDS. However, there comes a point when the
managing the cfengine configs becomes a burdensome and incredibly unmanageable.
I mean, sure, I know what they do. How will any of my co-workers understand
them? After several colleagues recommending
Puppet, I hesitantly began the slow, brain fscking
- Understanding exactly what I had accomplished with cfEngine.
- Understanding Ruby (ugh, I'm so thankful for Perl)
- Understanding how to express my cfengine feelings in a way Puppet will understand without hurting it's feelings