Splunk Integration With Xen

I've been dying to get my hands on a Fedora Core 5 for some time now. After a day of downloading, today I finally got to run my first FC 5 install. As expected, everything just works. I used CDs for my first install, but quickly setup the standard Red Hat/Fedora tree on my kickstart server. I'm glad I did.

The main draw for me with FC 5 is the integration with Xen. Up until now, I've had to use the live CD to try out Xen because I couldn't justify the time patching and compiling kernels to run inside Xen. With FC 5, all that work has disappeared. Here's a very good doc describing how to get started:

http://www.fedoraproject.org/wiki/FedoraXenQuickstartFC5

Now, I had one limitation on my development box, namely that it didn't have any outbound access to the 'net. That's okay, I setup squid on my workstation and was able to move on to configuring yum. I'm not sure what the problem was, but yum kept complaining about not being able to reach its repositiories. Whatever, I would think a default config file ought to have good default vaules in there but I guess not. After whacking that into shape I was able to install Xen on top of my fresh vanilla FC5. A quick edit to /etc/grub.conf (while I was talking to my Mom I might add, "Hi Mom) and I was all set to create domUs.

FC 5 provides a script to setup new domains automatically. This is nice.

Here's how I created one of my domUs, called xendomain2:

/usr/sbin/xenguest-install.py -n xendomain2 -f /home/xen/xendomain2 -r 256 -l http://kickstart.priv.nuasis.com/kickstart/fedora/core/5/i386/os/

This is okay, but I want an automated way. I don't like dealing with anaconda, enkay?

xenguest-install.py -n xendomain4 -f /home/xen/xendomain4 -s 25 -r 256 -l http://kickstart.priv.nuasis.com/kickstart/fedora/core/5/i386/os -x ks=http://kickstart.priv.nuasis.com/kickstart/cfgs/xen.cfg

Pow!

Here's whats going on at the moment on my xen test box:

[root@dev3-rep01 ~]# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 995 2 r----- 1103.9
xendomain2 13 256 1 ------ 39.2
xendomain1 14 256 1 -b---- 12.9
xendomain3 20 256 1 -b---- 13.7
xendomain4 25 256 1 r----- 481.4

[root@localhost ~]# rpm -qi kernel-xenU
Name : kernel-xenU Relocations: (not relocatable)
Version : 2.6.15 Vendor: Red Hat, Inc.
Release : 1.2054_FC5 Build Date: Tue 14 Mar 2006 02:22:51 PM PST
Install Date: Fri 31 Mar 2006 12:15:31 AM PST Build Host: hs20-bc1-3.build.redhat.com
Group : System Environment/Kernel Source RPM: kernel-2.6.15-1.2054_FC5.src.rpm
Size : 13268213 License: GPLv2
Signature : DSA/SHA1, Tue 14 Mar 2006 03:25:11 PM PST, Key ID b44269d04f2a6fd2
Packager : Red Hat, Inc.
Summary : The Linux kernel compiled for unprivileged Xen guest VMs
Description :
This package includes a version of the Linux kernel which
runs in Xen unprivileged guest VMs. This should be installed
both inside the unprivileged guest (for the modules) and in
the guest0 domain.


All above OSs are FC5 at the moment. I'm cool with that until I can figure out how to patch/install RHEL 3/RHEL 4 ontop of Xen on FC5.

I noticed some weirdness with ganglia, my open source cluster monitoring tool during and after the initial xendomain1 install. This is not surprising given all the virtual network and mac address forgery going on to support nesting OSs like this.

So, on to Splunk. I'm running v1.2.4 (thanks guys, for the quick turnaround on the installer). I wanted to capture these new logs coming out of the Xen install and configuration. I decidied to streamline my syslog-ng configuration on my splunk box a bit. Until now, each time I would add new machine, the syslogs wold end up under /var/log/remote-syslog-ng/$HOST/* where $HOST was the hostname/IP of the sending system. This is great for keeping logs seperate but sucks when it comes to modifying splunk configuration each time. So, I simplified my syslog-ng and have one "melting pot" for all my remote syslogs, regardless of originating system.

A few changes to my config and I was ready to restart syslog-ng.

That was fun!

Sphere: Related Content

Realer than real-deal Holyfield

This is the best article about the film I just saw Awesome, I Fucking Shot that.




I like wired, so I'm glad to see they covered this in the most detailed fashion. Interviewing MCA, aka Cap't Crunch, gives the story if full effect.

http://www.wired.com/wired/archive/14.04/play.html?pg=4

There were some other really cool effects that Yauch fails to mention. My favorites were the bass boom on Paul Revere and the total color wash out for a black and while nearly comic book effect.

And one more article, this one from the Sundance Review.

http://www.cinematical.com/2006/01/21/sundance-review-awesome-i-fuckin-shot-that/

Sphere: Related Content

Sourcefire Network Security - News & Events

I just got a very interesting email about the merger of Sourcefire and Checkpoint.

From: Jennifer Steffens
Update on Sourcefire Acquisition
2006-03-23 18:55

Hi Everyone,

We wanted to make everyone aware that Check Point and Sourcefire
withdrew their application today after carefully considering the
complexities of the CFIUS process, the lengthy ongoing delays in the
CFIUS process, and the current climate for international acquisition.

There is a press release with further details available at
http://www.sourcefire.com/news/press_releases/pr-13.html.

This in no way changes our commitment to the Snort technology or
community. If you have any questions, please let me know.

Thanks,
Jennifer



--
Jennifer S. Steffens
Director, Product Management - Snort
Sourcefire - Security for the Real World
W: 410.423.1930 | C: 202.409.7707
www.sourcefire.com | www.snort.org

Sourcefire Network Security - News & Events

Reading the above email and the corresponding press release,
it seems the reason the merger was called of was because of
complexities in the Committee on Foreign Investment in the
United States(CFIUS).

Here's another link on the aborted merger

http://www.informationweek.com/news/showArticle.jhtml;jsessionid=GCGILUQ1SVFLOQSNDBECKH0CJUMEKJVN?articleID=183702698

Sphere: Related Content

MercuryNews.com | 03/22/2006 | Dell buys Alienware, and AMD gets a foothold

MercuryNews.com | 03/22/2006 | Dell buys Alienware, and AMD gets a foothold

Sphere: Related Content

Network Monitoring Tools

Network Monitoring Tools

Sphere: Related Content

Hacking the Hack of Hacking Red Hat Kickstart

O'Reilly publishing has released a series of books called the Hacks Series. They define hack as "A clever solution to an interesting problem."

I'm a big fan.

I've read:

Google Hacks
Linux Server Hacks
Linux Server Hacks, Volume Two (Electric Boogaloo)
Mind Performance Hacks (just bought this one yesterday in fact)
Network Security Hacks

They're all really cool and eminently useful books.

Last Friday, I was assigned the task of creating a single CD based install of Red Hat Enterprise Linux for the purpose of unattended installation (Kickstart). This is opposed to the option of physically swapping all four CDs as released by Red Hat. Until now, we, and our customers, had been leveraging Kickstart using the stock RHEL Update 6 CD 1 to do network based installations. In my opinion, going over the network is more elegant than hacking up a custom CD any day. However, our "Partner" got our "Marketing Department" to agree that we would provide CD media for the entire installation of our product. As such, the network approache, while technically greatly superior, would have to take a back seat.

My starting point was an excellent article written by Brett Schwarz for Linux Journal. Here's the article link:

http://www.linuxjournal.com/article/6473

This article, unfortunately, is based on Red Hat 8.0, which is now obsoleted and past its end of life. Luckily, other Red Hat Enterprise customers have updated the doc with their tweaks to support all the latest RHEL distros. In this post, I will consolidate all the work done previously by Brett and add in the fixes/tweaks as added by the community. This follows the O'Reilly approach covered in their Hacks series "Hacking the Hack."

As Slick Rick says, "Here we Go."

First, get your hands on the 4 CDs for the RHEL release in question. For me, this was RHEL ES3 Update 6. Here are the CDs and their md5sums. (Yes, I know RHEL 3 Update 7 is out, so is RHEL 4 Update 3, and Fedora Core 5.) Again, full credit to Brett for all of these steps. They've just been tweaked as necessary to support newer RH releases and documented end-to-end here.

[root@kickstart iso]# ls -lah rhel-3-u6-i386-es-disc*
-rw-r--r-- 1 root root 152M Sep 21 11:54 rhel-3-u6-i386-es-disc1.iso
-rw-r--r-- 1 root root 627M Sep 21 11:43 rhel-3-u6-i386-es-disc2.iso
-rw-r--r-- 1 root root 637M Sep 21 11:48 rhel-3-u6-i386-es-disc3.iso
-rw-r--r-- 1 root root 282M Dec 6 16:38 rhel-3-u6-i386-es-disc4.iso

[root@kickstart iso]# md5sum rhel-3-u6-i386-es-disc*
2a695a0dc773b2172b35f8164b10f2f3 rhel-3-u6-i386-es-disc1.iso
68e7b2f34cb1903c24da02e25bcf5462 rhel-3-u6-i386-es-disc2.iso
8aa48608434065fb481d462d8495583c rhel-3-u6-i386-es-disc3.iso
3b35b450ecec27c5a9c63300f7518d3f rhel-3-u6-i386-es-disc4.iso

[root@kickstart iso]# mkdir Update6
[root@kickstart iso]# mkdir -p CD{1,2,3,4}
[root@kickstart iso]# mkdir onecd

Let's mount these badboys, and I don't want to hear any shit about the FHS right now either.

mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc1.iso /kickstart/ES3/Update6/CD1/
mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc2.iso /kickstart/ES3/Update6/CD2/
mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc3.iso /kickstart/ES3/Update6/CD3/
mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc4.iso /kickstart/ES3/Update6/CD4/

Copy the RPMS:

cp -a CD1/* onecd/
cp -a CD{2,3,4}/RedHat/RPMS/* onecd/RedHat/RPMS/

Now the tricky part, coming up with a complete set of RPMs that fits on one CD and is internally consistent. Let's use Brett's python scripts to get this started.

cd /kickstart/ES3/Update6/onecd/

getGroupPkgs.py comps.xml > /kickstart/ES3/Update6/pkglist
syncRpms.py onecd/RedHat/RPMS/ pkglist > pkgs_rem

(Note, I had to make a few changes to the syncRPMS.py script to reflect the newer arcitecture) Here's the modified script:

#!/usr/bin/python

#
# Removes packages that are not part of a package list from
# a given directory. This is used to remove RPMs that are
# not needed in a custom distro. The package list specified
# is just a text file with a list of package names, one per
# line.
#
# syncRpms.py /some/path/to/rpms/ /tmp/pkglist
#
# Copyright (C) 2003 Brett Schwarz (brett_schwarz@yahoo.com)
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
# 02111-1307 USA
#
# Version History
# ---------------
# 0.2 08-04-2003 Added fix from Alain Tauch for accepting
# /path instead of /path/
# 0.1 22-03-2003 Original release

import rpm
import sys, os, glob

if len(sys.argv) != 3:
sys.stderr.write("Usage\n")
sys.stderr.write("%s \n" % (sys.argv[0],))
sys.exit(1)

tgtdir = sys.argv[1]
pkglist = sys.argv[2]

#
# get pkg name to file name mapping
#
name2file = {}
ts = rpm.TransactionSet("", rpm._RPMVSF_NOSIGNATURES)
for fname in glob.glob(tgtdir + '/*.rpm'):
fd = os.open(fname, os.O_RDONLY)
hdr = ts.hdrFromFdno(fd)
name2file[hdr[rpm.RPMTAG_NAME]] = fname
os.close(fd)

#
# Read in packages from package list
#
fd = open(pkglist, "r")
pkgs = {}
for l in fd.readlines():
if l[-1]=='\n':
l = l[:-1]
pkgs[l] = 1

fd.close()

#
# Remove unwanted packages
#
for n, f in name2file.items():
if not pkgs.has_key(n):
os.remove(f)
print n

#
# Check to see if there are pkgs not in tgt dir
#
for p in pkgs.keys():
if not name2file.has_key(p):
print "Package not in tgt dir: ", p

OK. Now to test the resultant set. Again, cool trick Brett.

mkdir /tmp/testdb
rpm --initdb --dbpath /tmp/testdb
rpm --test --dbpath /tmp/testdb -Uvh *.rpm

For whatever reason, Brett's script didn't do a very good job of gettin the right set together. No matter, the rpm command tells you about failed dependencies so all you have to do is go locate the missing RPMs and copy them into the oncecd tree. I ended up building up a third file called pkgsadd and then ran:

for i in $(cat pkgsadd); do src=$(find CD* -name $i); /bin/cp -f $src onecd/RedHat/RPMS/; done

Eventually, the set was consistent.
[root@kickstart RPMS]# rpm --test --dbpath /tmp/testdb -Uvh *.rpm
warning: acl-2.2.3-1.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
warning: package glibc = 2.3.2-95.37 was already added, replacing with glibc <= 2.3.2-95.37 warning: package kernel = 2.4.21-37.EL was already added, replacing with kernel <= 2.4.21-37.EL warning: package kernel-smp = 2.4.21-37.EL was already added, replacing with kernel-smp <= 2.4.21-37.EL Preparing... ########################################### [100%]

One more test, for good measure: rpm -K *.rpm | grep "NOT *OK" Now, here's where my steps differed from the initial doc.

Pay attention.

First, you need anaconda and anaconda-devel.

up2date anaconda anaconda-runtime
export PYTHONPATH=/usr/lib/anaconda
/usr/lib/anaconda-runtime/pkgorder /kickstart/ES3/Update6/onecd/ i386 > /kickstart/ES3/Update6/onecd/RedHat/base/pkgorder
/usr/lib/anaconda-runtime/genhdlist --withnumbers --fileorder /kickstart/ES3/Update6/onecd/RedHat/base/pkgorder --hdlist /kickstart/ES3/Update6/onecd/RedHat/base/hdlist /kickstart/ES3/Update6/onecd/

Now, I'm going to add all the ks.cfg files to the CD repository.

[root@kickstart onecd]# http://satellite.priv.nuasis.com/kickstart/ks/label/ncc-3.0
[root@kickstart onecd]# wget http://satellite.priv.nuasis.com/kickstart/ks/label/ncc-3.0-ide
[root@kickstart onecd]# wget http://satellite.priv.nuasis.com/kickstart/ks/label/ncc-3.0-md

This almost bit me:

[root@kickstart CD1]# cp .discinfo /kickstart/ES3/Update6/onecd/

One more fix I had to do. My ks.cfgs out there on the satellite serve all point to a network based install. To make this new kickstart go off of the cdrom, I need to point to cdrom instead of url inside each of ncc-3.0, ncc-3.0-ide, and ncc-3.0-md.

The big step:

[root@kickstart root]# mkisofs -r -T -J -V "ncc-3.0-rhel-3-u6-i386-es.t1" -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -v -o /kickstart/iso/ncc-3.0-rhel-3-u6-i386-es.t1.iso /kickstart/ES3/Update6/onecd

154528 extents written (301 MB)

301 MB, not bad.
[root@kickstart root]# /usr/lib/anaconda-runtime/implantisomd5 /kickstart/iso/ncc-3.0-rhel-3-u6-i386-es.t1.iso
Inserting md5sum into iso image...
md5 = 12bf4c82d5d59b854e00d343b02d7cc6
Setting supported flag to 0

Cool.

Sphere: Related Content

Announcing the release of Fedora Core 5

Announcing the release of Fedora Core 5

Hi, my name is Fedora Core "Bordeaux", and today I am 5.  When I
turned 4 last year, they got a funny salesman to talk about me like I
was a toy.  I like toys.  But today Teacher said I am a big kid, and I
should talk about myself.  I can do lots of big kid stuff now, and
everyone tells me that I play really well with all the other kids in
class, even the ones who are mean like bullies.  I always try and
share, which is what Teacher says is the best thing.

Sharing is a really good thing to do.  I like to share all my toys and
books and stuff with all the other kids so that they can play with
them and even learn things, too.  I have a lot of neat stuff to share,
so I made this cool list to tell you all about it.  Teacher gave me
some things to put in the list, but since I'm a big kid I think I
should get to say what I want, too.

* GNOME 2.14

"Improved speed and usability, and new and better features for power
management, file sharing, user help, system administration,
teleconferencing, picture browsing, and networking."  I think it's
easy enough for even my little brother to use, but Mommy and Daddy
like it a lot too.

* OpenOffice 2.0.2

"Enhanced productivity features, extensive compatibility with other
popular office software, database connectivity functions, and improved
use of system libraries for faster loading and better responsiveness
overall."  Mommy says she can make slide shows for work even easier
than before, and Daddy uses it to organizes all our books, music, and
movies in databases.

* KDE 3.5.1

"New and exciting features for users and developers, including stylish
and attractive applets, educational and entertaining games, incredibly
standards-compliant Internet tools, and enhanced multimedia and
usability."  Plus it looks REALLY cool, especially when my big sister
fixes it just the way she likes.

* Mono

"Support for .NET means a new generation of dynamic and powerful
cross-platform applications, with some already included, such as
Tomboy for note-taking, F-Spot for photo management, and Beagle for
content searching and indexing."  See, I told you I'm really good at
sharing!  Plus, I don't ever have a problem remembering or finding
things, like pictures, homework, or messages.  Mommy says she's really
proud of me.

* Yum-based Package Tools

"Installation and software management tools are all based on the
flexible and powerful yum utility for easy selection and upgrade of
new and existing software."  And soon, I'll even be able to get my own
special software at install time!

* Xen Virtualization

"The best framework yet for installation, management, migration, and
monitoring of software-based virtual domains, allowing system owners
to effectively leverage and force-multiply existing hardware for
maximum efficiency, scalability, redundancy, and flexibility."
Teacher says I get more work done faster than any other kid in our
class, too.

* Apache HTTP Server 2.2

"Enhanced authentication, database support, proxy facilities, and
content filtering, all built on the most stable and customizable
platform for Web services."

* Enhanced Security

"Support for SELinux binary policy modules means that users and
developers can now ship their own specialized policies with affected
software.  GCC's best of breed features, such as stack protection, NX,
PIE, and compile-time buffer checks, pile on additional layers of
assurance to effectively block illegal ingress."  I know all my safety
rules, and Mommy and Daddy say that it's really, REALLY hard for
monsters to get in our house!


So at school, I got awards for "Best Attendance," "Teacher's Pet,"
"Honor Roll," and "Good Manners," plus Teacher lets me stay after to
clean the erasers until Daddy comes to pick me up.  He always tells me
how good I am compared to the other kids he knows and that he's really
proud of me.  He says it's because I'm 100% free and open source, and
because I always try to do the right thing just like he and Mommy
taught me.

I told him I didn't just learn it from him, but from all the nice
people at Red Hat and the Fedora community, who teach me new things
all the time.  They're all different kinds of people -- users,
developers, writers, translators, testers, editors, and so many other
things I can't even write it all.  They're the people who make it
possible to build a complete Linux platform from open source software.

But they say we're not done yet, and we can ALWAYS use more friends to
help, so if you think this sounds neat, you should definitely visit:

* http://fedoraproject.org/wiki/HelpWanted

All right, I have to go now.  There's always more things to learn and
do and I am a very, extremely busy kid.  'Bye!

- - -

Fedora is a set of projects sponsored by Red Hat and guided by the
contributors.  These projects are developed by a large community of
people who strive to provide and maintain the very best in free, open
source software and standards. Fedora Core, the central Fedora project,
is an operating system and platform, based on Linux, that is always free
for anyone to use, modify, and distribute, now and forever.

Fedora Core 5 is available at absolutely no cost.  To download it:

VIA BITTORRENT (RECOMMENDED):

* http://torrent.fedoraproject.org/bordeaux-binary-i386.torrent
* http://torrent.fedoraproject.org/bordeaux-binary-x86_64.torrent
* http://torrent.fedoraproject.org/bordeaux-binary-ppc.torrent

For DVD and other formats, refer to http://torrent.fedoraproject.org/

If you run an earlier version of Fedora Core, you can get BitTorrent
from Fedora Extras.  If you are using another platform, you can get
BitTorrent at:

* http://www.bittorrent.com/

VIA WEB:

Visit the main Fedora download site listed below.  You will be automatically
redirected to a mirror of Fedora Core 5.  There may be delays due to site
congestion, especially in the days immediately following the release, so
BitTorrent is recommended instead.

* http://download.fedora.redhat.com/pub/fedora/linux/core/5


VIA CD/DVD:

Visit the following site for a list of vendors of Fedora Core CD/DVD
media:

* http://fedoraproject.org/wiki/Distribution/OnlineVendors

Sphere: Related Content

Argogroup : Home

Argogroup : Home

Sphere: Related Content

Advertising: mobile marketing conversion rates

Advertising: mobile marketing conversion rates

Sphere: Related Content

MMA Global - 89% of Major Brands Planning to Market via Mobile Phones by 2008; Mobile Marketing to Accelerate with More Than Half of Brands Planning t

MMA Global - 89% of Major Brands Planning to Market via Mobile Phones by 2008; Mobile Marketing to Accelerate with More Than Half of Brands Planning to Spend up to 25% of Marketing Budget (Airwide Solutions)

Sphere: Related Content

Syslog-ng

shib·bo·leth (shĭb'ə-lĭth, -lĕth')
n.

1. A word or pronunciation that distinguishes people of one group or class from those of another.
2. 1. A word or phrase identified with a particular group or cause; a catchword.
   2. A commonplace saying or idea.
3. A custom or practice that betrays one as an outsider.

http://www.campin.net/newlogcheck.html

http://www.oreilly.com/catalog/bssrvrlnx/chapter/ch10.pdf

http://www.balabit.com/products/syslog_ng/

http://dmourati.blogspot.com/2006/02/splunk-question-and-answer.html

http://dmourati.blogspot.com/2006/01/splunk-and-syslog-ng.html

Sphere: Related Content

Splunk Integration With OpenSSH

This blog post covers Splunk Integration with OpenSSH. OpenSSH is the lifeblood of any distributed Linux system. I practically live inside and SSH session. During any given day, I may SSH into 20 different system all around the world. As such, it is important to keep track of what SSH is doing. What better way than with Splunk?

This integration actually picks up after some work I did earlier integrating Splunk with Syslog-ng. For more information on how I'm setup, please read my earlier post

Splunk and syslog-ng

Now that I have remote syslog enabled, I can do all kinds of cool stuff. This is especially useful for troubleshooting large networks of interconnected systems where SSH is undoubtedly already in play. My syslog-ng configuration is already being processed by Splunks tailingprocesser which is looking at all the stock syslog-ng files plus per host syslog-ng directories for remote syslog. Here' s a look at my central syslog-ng configuration:

[root@demetri05 log]# find remote-syslog-ng/
remote-syslog-ng/
remote-syslog-ng/demetri04
remote-syslog-ng/demetri04/secure
remote-syslog-ng/demetri04/messages
remote-syslog-ng/demetri04/cron
remote-syslog-ng/demetri04/boot.log
remote-syslog-ng/demetri04/maillog
remote-syslog-ng/demetri04/kern
remote-syslog-ng/demetri05
remote-syslog-ng/demetri05/boot.log
remote-syslog-ng/demetri05/cron
remote-syslog-ng/demetri05/secure
remote-syslog-ng/demetri05/messages
remote-syslog-ng/demetri05/maillog
remote-syslog-ng/demetri05/kern
remote-syslog-ng/splunk
remote-syslog-ng/splunk/demetri04-messages
remote-syslog-ng/splunk/demetri05-messges


To get a sense of just how pervasive SSH is in my environment, take a look at the following splunk. This shows that in the past day, there are over 10,000 events relating to SSH. Wow. So far I've uploaded all the standard SSH messages I could generate. That turned out to be around 20 types or so. Now, I'm going to make some changes to my SSH server to increase its logging level. Take a look at the /etc/ssh/sshd_config file for reference:

# Logging
#obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

You'll notice a large uptick in the number of events starting in the 12:00 PM hour today. That's becaue I've changed the LOGLEVEL configuration in my /etc/ssh/sshd_config file to DEBUG. WARNING, don't do this on any system with real users. According to the manpage "Logging with a DEBUG level violates the privacy of users and is not recommended." This is my system and I'm the only user so I can proceed but I won't leave this in place beyond my work today for reasons of privacy and also to keep my log files manageable.

Here's a look at the type of events generated by the inceased LOGLEVL in SSHD. This time I'm splunking for sshd debug1.
That's kind of cool, actually. Now I have a whole set of new event types to upload to SplunkBase. Again, the fact remains that real systems shouldn't be configured with this level of logging for SSH but I'll make sure to note that in all my event descriptions once they're uploaded. I'm going to simplify my work by sorting the events into event types and then uploading each event type on up to SplunkBase for integration. That way I'll be sure to get all the event types up there without worrying about duplicating log submissions. At the moment, I see 36 event types related to SSH and the new DEBUG loglevel I've put in place.
I'm drilling down on each of these event types, then uploading the corresponding log entry up to splunk base one at a time. It seems like this is uncharted territory as far as SplunkBase is concerned. Each of these event types are new which is cool.

As I go, I'm keeping an eye on the total number of event types I've submitted to SplunkBase by watching my splunk user page here:

http://www.splunk.com/user/dmourati

At the moment, I'm at 383 total event types submitted and counting. Completing this SSH debug logging should push the total above 400 types.

Now I'm going to reset the LOGLEVL on my SSH configuration to turn off all this DEBUG level logging.
That in turn generated a bunch more event types moving the total from 36 to 43. Hmm. Let me add those for good measure. The new event types are easy enough to spot because they each have only one associated event.

Here's a link to the last event type from my SSH DEBUG testing:

http://www.splunk.com/event/SP-CAAAC5H

Sphere: Related Content

Splunk Integration with Firefox

OK, so this isn't much of an integration, per se. But a nice tip nonetheless.

I've been doing a lot of splunking lately. Here's one tip I wish I had back when I got started.

First, upgrade to Fifrefox 1.5. I downloaded a linux tarball here:

http://www.mozilla.com/products/download.html?product=firefox-1.5.0.1&os=linux&lang=en-US

Go to Edit->Prefences
Select Tabbed Browsing
Select "New Tab" from the "Load Web Browswer Links into a:" pulldown.

Note, under windows, the option was slightly different. There I had to
Go to Tools->Options->Tabs

Force Links That Open New Windows in:
* a new tab

Next, for either platform, open the following url in your location bar:

about:config



Look for the preference name

browser.link.open_newwindow.restriction

Set it to 0.

Now when the JavaScript windows for SplunkBase pop up in the Check splunk.com links, they resulting page will open in a new tab instead of a new window.


One more tweak was to overcome irritating messages about unresponsive scripts. I used about:config again this time to set

dom.max_script_run_time

to 30

Other users have reported a similar need to tweak the script runtime max for Yahoo sites as well as gmail.

Here's more info on the above:

http://www.mozilla.org/support/firefox/tips

Nice.

Sphere: Related Content

$1000 Laptop

"Hardware is a small part of the cost" of providing computing capabilities, he said, adding that the big costs come from network connectivity, applications and support.




Bill Gates mocks MIT's $100 laptop project - Yahoo! News






You know what I like about Microsoft?

Nothing.

You whow what I like about Bill Gates?

Nothing.

Yesterday I could have appluaded him for his humanitarian efforts, cover of Time, all that jazz. But now? What kind of sick bastard endows a foundation with $29 million for global health and learning then turns around and slams the $100 laptop?

It's a project from MIT. Oh, yea, I guess those guys don't know what they are talking about.

http://laptop.media.mit.edu/

$100 laption, pshaw.

No, what you really need is my $1000 "ultra mobile computer." Just say that with your geekiest voice "ULTRAMOBILE COMPUTER!! UBC!! NNNAAAAA NAAAA"

What Is the Ultra-Mobile PC you ask?

The Ultra-Mobile PC is a new kind of computer. It combines the power of Windows XP with mobile-ready technologies that make it easy to access and use your software on the go.

With small, lightweight, carry-everywhere hardware designs, you can connect and communicate, accomplish any task anywhere and at any time, and be entertained and informed wherever life takes you.

Would whoever wrote that please stand up and report for beating with a wiffle ball bat?

Thank you.

So, what OS do you suppose runs on the MIT project?

Ask Nicholas Negroponte.

The machines, which will run a version of the Linux operating system, will also include other applications, some developed by MIT researchers, as well as country-specific software. "Software has gotten too fat and unreliable, so we started with Linux," he said.

Hmm. Sounds about right.

Sphere: Related Content

Phone, media firms see mobile ads soon - Yahoo! News

Phone, media firms see mobile ads soon - Yahoo! News

Sphere: Related Content

Is Dell buying Alienware? - Engadget

Is Dell buying Alienware? - Engadget

Sphere: Related Content

Systems architect - Wikipedia, the free encyclopedia

Systems architect - Wikipedia, the free encyclopedia

Sphere: Related Content

Nuasis NuContact Center 3.0

Here are some press links around the recent release of the Nuasis NuContact Center Version 3.0.

CRM Magazine/Destination CRM

Article headline: All the Talk at VoiceCon: The contact center market continues its migration to IP as vendors make a host of IP-related announcements
http://www.destinationcrm.com/articles/default.asp?ArticleID=5894

NOTE: Nuasis spoke to writer, Coreen Bailor, and the analyst who is quoted, Joe Outlaw. While the article states that there were a "slew of product enhancements, new services and rebranding" at VoiceCon, only Nuasis, Cisco and Avaya are referenced in the article.

Destination CRM is the CRM industry's largest, most successful website which includes the newsletter, CRM eWeekly and the digital version of CRM Magazine, both with 95,000 opt-in subscribers. Readership is mixed between IT management, corporate management and sales/marketing/services.

VoIPLoop: IP Telephony for the Enterprise

Article headline: VoiceCon Spring '06: Interview with Kevin McPartlan, Vice President of Nuasis

http://www.voiploop.com/index.php?option=com_content&task=view&id=899&Itemid=1

NOTE: Before VoiceCon, Nuasis spoke to Alex Dunne, editor of VoIPLoop - the written interview is the link above.

Podcast headline: VoIP Security with Brendan Ziolo of Sipera Systems and IP Contact Center Issues with Kevin McPartlan of Nuasis

http://voicecon.libsyn.com/index.php?post_id=66465

NOTE: During VoiceCon, Nuasis spoke to Eric Krapf, VoiceCon conference co-chair and editorial director of BCR Magazine. Eric conducted a podcast interview with Kevin McPartlan.

Blog headline: VoiceCon Conference within a Conference Preview

http://www.voiploop.com/index.php?option=com_content&task=view&id=887&Itemid=34

NOTE - Nuasis talked to Sheila McGee-Smith before VoiceCon. The panel that Sheila discusses in her blog had a standing-room only audience and a standing-room only crowd outside the meeting room who listened to the panel discussion through the room's open door. Panel participants were executives from Avaya, Cisco, Genesys, Nortel and Nuasis' Kevin McPartlan.

VoIPLoop is an online interactive forum for analysts, press, end-users and vendors. It is a CMP Online Publication that was launched in late 2005.

Customer Interaction Solutions/TMCNet

Article headline: Nuasis Says NuContact Center 3.0 Now Available

http://news.tmcnet.com/news/2006/03/07/1434393.htm

Article headline: First Coffee - Cingular Announces Video, Nuasis Announces NuContact Center 3.0, Parature Announces Another Customer

http://www.tmcnet.com/usubmit/2006/03/07/1434332.htm

NOTE: Also mentioned in this article is Nuasis' newest customer, Southwest Gas.

TMCnet.com is the online site that accompanies TMC's four print publications. TMCnet.com imparts information that helps readers become communications and call center technology advocates who make purchasing decisions and participants who will make these technologies work in any given implementation -- whether it be call centers, the central office or a corporate enterprise. The site currently averages over 7 million page views and reaches on average 650,000 unique visitors per month.

CRMAdvocate - Everything CRM - Read Less. Know More.

TODAY'S NEWS

Nuasis announced the availability of NuContact Center 3.0 software. This release supports agents located anywhere, higher system scalability, and advanced conditional routing based on customer intelligence and business analytics. NOTE: Nuasis' announcement of 3.0 was the lead story in the March 7 edition of CRMAdvocate.

CRMAdvocate is the definitive and most complete source of information concerning technologies, strategies, and trends for Customer Relationship Management (CRM) and Contact Centers. Providing real-time and on-demand industry news, webcasts, case studies, white papers, research, and product information, CRMAdvocate allows end-users, editors, research analysts, system integrators, consultants and others to stay current on industry trends. Owned by RealMarket, CRMAdvocate is headquartered in Indianapolis, Indiana.

Call Center Magazine/CommWeb and Communications Convergence.com

Article Headline: Nuasis NuContact Center 3.0: The Updated System Supports Remote Agents and Has Greater Scalability

http://www.cconvergence.com/shared/article/showArticle.jhtml?articleId=181501745

CommWeb and its sister site, CommunicationsConvergence.com (Call Center Magazine online) reach the people who make the business communication technology buying decisions.CommWeb/cConvergence's editorial provides end users and the telephony resellers that serve them the information they need to get their jobs done.

CMP Call Center Network

Sphere: Related Content

Splunk Integration with Weblogic

Continuing on a theme, this next installment details a new project to integrate Splunk with Weblogic, the J2EE application server. By now you already know about splunk, so let me say a few words about weblogic. Weblogic is billed as "
The most powerful, reliable release of the world's leading J2EE application server is the ideal foundation for building SOAs." What the heck does this mean? Well, weblogic is a J2EE server which means it follows the standard as formalized and defined by Sun. In a previous insallment, I covered a Splulnk Integration with JBoss, which is another J2EE compliant server. The third leading J2EE server is Websphere from IBM. I spent the better part of last week working directly with IBM on a porting project from Linux on x86 architecture to Linux on IBM's POWER architecture, namely the iSeries and I5/0S (fka AS/400). That project involves a port of JBoss at the moment, but I could see how Websphere, with its native integration to I5/0S and native J2SDK might be a better option for ease of deployment and native compatibility. In summary, there are three major J2EE application servers out there at the moment, in this installment we'll cover integrating splunk with a J2EE server, this time Weblogic.

As part of this project, I'll also cover the upgrade of splunk 1.1 to splunk 1.2. Upgrades are supported as of splunk 1.2 and I've already been contacted by a splunker to verify that I'm clear for launch. Thanks Ariel!

Here's the relevant section of the docs, Updating Prior Versions
. The trick seems to be that the configurations done the etc subdirectory get saved as etc.bak.

Here's a look at the demetri05 machine where I'll be working today:

[root@demetri05 root]# ls -l splunk-*
-rwxrwxr-x 1 root root 24451062 Jan 6 11:05 splunk-Professional-1.1-linux-installer.bin
-rwxrwxr-x 1 root root 23916072 Jan 4 17:10 splunk-Server-1.1-linux-installer.bin
-rw-r--r-- 1 root root 21483701 Mar 7 22:16 splunk-Server-1.2-linux-installer.bin

So, looks like I need to turn on execute bits for the new splunk 1.2 installer.bin. Repeat gripe, I need an RPM for splunk and a fully non-interactive install. Again RPM is *the* standard for Linux so this .bin stuff has got to go.

[root@demetri05 root]# chmod a+x splunk-Server-1.2-linux-installer.bin

So invoking the installer like so:

[root@demetri05 root]# ./splunk-Server-1.2-linux-installer.bin

I'm presented with a license agreement and now asked to specify the target installation directory, I'll keep with /opt.

Now I'm presented with the following information, it looks like the installer has detected my existing install. (One more thing, the server has been stopped prior to this point, not sure if that is required but usually a good idea just to be nice.)

Installation Directory [/opt]:

Info: We found a previous Splunk installation. We will retain all indexed data,
user accounts, Saved and Live Splunks, event type tags, and custom source type
names. The configuration files from the previous installation will be saved in
the directory /opt/etc.bak in case you still need them.

And yes, I still need those config files so I'm glad they will be saved. This will save me a lot of trouble going forward.

I select enter and move on.

I'm presented with a number of options concerning ports to run splunk on and directories to use, I'll keep with the defaults here as I have no reason to do otherwise. One other configuration option is running multiple splunk servers on the same machine side-by-side. If you were to go that route, give some careful thought to port allocations and directory structures as you can easily overwhelm the system ,and its administrator, and cause all types of seemingly weird behavior. Like Professor Griff says, "London England, Consider Yourselves Warned!"

A couple more enters and here we go:

Please wait while Setup installs the Splunk Server on your host.

Installing
0% ______________ 50% ______________ 100%
############

Ahh, the sweet sign of success:

Installation successful. See the README.txt file in your install directory.

Let's take a look at the new install:

[root@demetri05 opt]# ls
customer.xml home ncclogs solipsa.com splunk
[root@demetri05 opt]# cd splunk/
[root@demetri05 splunk]# ls
bin etc etc.bak0 lib openssl README.txt sbin share uninstall var

As we said, there is a bak directory, it's named etc.bak0. This is to allow for an upgrade without destroying my initial config. Let's look for config files.

[root@demetri05 splunk]# find etc.bak0/ -name config.xml
etc.bak0/modules/historyprocessor/config.xml
etc.bak0/modules/tailingprocessor/config.xml
etc.bak0/modules/fifoInput/config.xml
etc.bak0/modules/statprocessor/config.xml
etc.bak0/modules/directorymonitor/config.xml

I'm tempted to copy them over to the real etc for preservation but unsure whether there have been any other config file format changes. One file I worked with last time was modules/tailingprocessor/config.xml. Lets see a diff against the new file. So the diff is fairly large and mostly due to a syslog-ng integration I've done unrelated to JBoss or Weblogic. Otherwise, the diff looks okay at first blush. I'll try copying this file over to the new install, making a backup of the v1.2 config.xml for safekeeping.

[root@demetri05 splunk]# cp etc/modules/tailingprocessor/config.xml etc/modules/tailingprocessor/config.xml.orig
[root@demetri05 splunk]# cp etc.bak0/modules/tailingprocessor/config.xml etc/modules/tailingprocessor/config.xml
cp: overwrite `etc/modules/tailingprocessor/config.xml'? y

Okay, that's done. Let's take another look at the preserved etc.bak0 directory to see if any other files should be copied over. I've found this file, which looks interesting.

etc.bak0/custom-typers/weblogic.typer.xml

There's also one in the current install's etc directory. What's the diff?

Huge, again. I'll keep the new one and hope that someone has made some improvements to the typing.

So, I'll now try starting up the new server for the first time. If this doesn't work, my plan is to backout the config.xml change to the tailingprocessor and start again.

The moment of truth:

[root@demetri05 splunk]# /etc/init.d/splunk start
== Checking prerequisites...
Version is Splunk Server
Checking http port [8000]: open
Checking https port [8001]: open
Checking mgmt port [8089]: open
Checking search port [9099]: open
== All checks passed
Starting splunkd [ OK ]

Success!

Now were cooking with fire. Here's the first look at a fully working new 1.2 front end.

Weblogic is available for download from the BEA site here:

12/16/2005. WebLogic Server 9.1

I had to create a user account and password and say that I was downloading Weblogic for an eval in order to proceed. I've selected the binary for Red Hat Enterprise Linux 3/4 and am now in the process of downloading it. It's about 350 MB of installer so either use a fat pipe or be prepared to wait.

Here are the install instructions for Weblogic Server 9.1.

Installation Guide


Reviewing the docs, Weblogic has a console mode installer for UNIX systems. I'll use that.

[root@demetri05 root]# chmod a+x server910_linux32.bin
[root@demetri05 root]# ./server910_linux32.bin

My first change was to move bea home from /root/bea to /opt/bea.

I then selected complete install.

I let the installer put down the Mercury profiling tool, though I don't plan to use it.

The product installation directory was "/opt/bea/weblogic91."

Here it goes:

<---------------------------------------------- BEA Installer - BEA Products ---------------------------------------------->

Installing files..

0% 25% 50% 75% 100%
[------------|------------|------------|------------]
[********

So, at the end of this process, Weblogic is installed under /opt/bea/weblogic. Great. Now, how do I start this sucker? The docs are silent on the matter on that pointing to a quickstart launcher that requires X. Okay, so now what.

Aha, here we go:

Starting an Administration Server with a Startup Script

So, I guess I need to use the Weblogic Scripting Tool or WLST.

I found a default template called medrec that I just started playing with.

Here's how I started it:

[root@demetri05 bin]# cd /opt/bea/weblogic91/samples/domains/medrec/bin
[root@demetri05 bin]# ./startWebLogic.sh

Let's configure Splunk's tailing log file processor to grab the log files for this weblogic domain.

Again, I editted /opt/splunk/etc/modules/tailingprocessor/config.xml and added a new stanza for the weblogic log file located at /opt/bea/weblogic91/samples/domains/medrec/servers/MedRecServer/logs/MedRecServer.log.

I found one more default server that Weblogic installed and added its log file to the config as well. That log file was:

/opt/bea/weblogic91/samples/domains/wl_server/servers/examplesServer/logs/wl_server.log

Both now appear on my Splunk front end GUI page which means this is working!

Now, I'm going into the Splunk GUI and uploading these event types to SplunkBase by clicking on the check splunk.com links.

Now that some of these logs are up on SplunkBase, you can start to see how this whole thing fits together. Here's an example type I've just uploaded:

SP-CAAACM6

For this example, I added two tags to this event. JDBC and MedRecServer.

Let's try searches for those two tags:


MedRecServer

JDBC

Cool.

Right now I'm just scanning through the EventTypes on my local Splunk server. I've got them sorted in reverse frequency so that I'm uploading the rarest event types first.

This time around, I've noticed two things about the EventTyping. First, Splunk detected this log file as Type weblogic_stdout. Second, Splunk seems to be auto-tagging. This is really cool. Saves me some typing later.

The other thing I'm doing is hanging some "meat" on each event type description. For now, its just the standard, weblogic log, see bea webpage for more infor kind of message. Later, folks with more detailed knowledge about each of these can come online and put some better information around each event type. For now, though, this is a good header.

So I've uploaded the bulk of my events to splunk base and editted them to taste.

I'm now up around 200 events submitted with the majority of those containing some significant detail.

Here's the latest:

http://www.splunk.com/event/SP-CAAACPW

There you have it, Splunk Integration With Weblogic

Sphere: Related Content

iMedia Connection: Mobile Video: Present and Future

Here's an article I found as part of my research into the mobile ASP space. It sounds promising, doesn't it?

iMedia Connection: Mobile Video: Present and Future

Sphere: Related Content

Open Season On Open Source?

There's an interesting article in BusinessWeek about the future of Open Source after Oracle's recent maneuvers with Innodb, MySQL, and JBoss. The cool thing is that the open source project the writer picked to anaylyze is none other than Nagios, one of my favorite tools.

Open Season On Open Source?

See also my earlier blog post about integrating Splunk with JBoss and the reference at the end to the Oracle situation.

Splunk Integration with JBoss

When I contacted Ethan Galstad to notify him of the article, he was kind enough to let me know that it is posted on the Nagios site under the propoganda section here:

Nagios Propaganda

He also posted a reply to a blog on InfoWorld here:

InfoWorld Blog

I tend to agree with Ethan on his main point that the Open Source community behind projects like Nagios and a corporation seeking to "buy in" to such a community have differing objectives and motivations. What still remains unclear to me, however, is to what extent dangling money in front of successful Open Source leaders will lead to a bastardization of the community. I'm hopeful that folks like Ethan will continue to show resolve and resist the temptation but I can also understand those who are willing to accept the risks of meddling in order to make a living off their hard work. Time will tell.

Sphere: Related Content

Contact Center News: VoIP, Nuasis, SureWest, Unity4, Digium Asterisk PBX

Contact Center News: VoIP, Nuasis, SureWest, Unity4, Digium Asterisk PBX

Sphere: Related Content