2011/02/09

Monitoring Tomcat with Jmxterm and Python

I needed to collect statistics related to connections in Tomcat. The goal was to use some command line tool to make dumping and post-processing of acquired data as simple as possible. It seems that jmxterm fitts well into these requirements. Following text represents short HOWTO for monitoring some of Tomcat's JMX attributes using jmxterm.
  1. Enable JMX in your Tomcat, e.g. by adding following parameters to its java command line.
            -Dsun.management.jmxremote -Dcom.sun.management.jmxremote.port=3993
            -Dcom.sun.management.jmxremote.ssl=false
            -Dcom.sun.management.jmxremote.authenticate=false
    As you see this settings turns off  both SSL and authentication, to make this simplem, although not secure.
    Jmxterm does not support SSL yet, password-based authentication in command open is supported (use help open in jmxterm shell for more info).
  2. Download jmxterm from http://sourceforge.net/projects/cyclops-group/files/jmxterm/
  3. Start in interactive mode to test commands:
    java -jar jmxterm-${version}-uber.jar
    (for ${version} substitute current version of jxmterm, e.g. "1.0-alpha-4")
  4. Run commands to get desired data
    1. open localhost:3993
    2. Restricts output of following commands to given domain - select domain: domain Catalina.
    3. Now you can list beans:
      beans
    4. now the domain is set and we get values of some attributes:
      • get -b Catalina:port=80,type=Connector acceptCount
        get -b Catalina:port=80,type=Connector acceptCount
        #mbean = Catalina:port=80,type=Connector:
        acceptCount = 100;
      • get -b name=http-80,type=ThreadPool maxThreads
        get -b name=http-80,type=ThreadPool maxThreads
        #mbean = Catalina:name=http-80,type=ThreadPool:
        maxThreads = 40;
      • get -b name=http-80,type=ThreadPool currentThreadCount
        get -b name=http-80,type=ThreadPool currentThreadCount
        #mbean = Catalina:name=http-80,type=ThreadPool:
        currentThreadCount = 15;
      • get -b name=http-80,type=ThreadPool currentThreadsBusy
        get -b name=http-80,type=ThreadPool currentThreadsBusy
        #mbean = Catalina:name=http-80,type=ThreadPool:
        currentThreadsBusy = 2;
      • get -b Catalina:host=localhost,path=${webapp_context},type=Manager activeSessions
        get -b Catalina:host=localhost,path=${webapp_context},type=Manager activeSessions
        #mbean = Catalina:host=localhost,path=${webapp_context},type=Manager:
        activeSessions = 3;
    5. Put commands in file and run this scripts in non-interactive way
      java -jar jmxterm-${version}-uber.jar -n
      or
      java -jar jmxterm-${version}-uber.jar -n -i jmx_script

      content of the file jmx_script:
      open localhost:3993
      get -d Catalina -b Catalina:port=80,type=Connector acceptCount
      get -d Catalina -b name=http-80,type=ThreadPool maxThreads
      get -d Catalina -b name=http-80,type=ThreadPool currentThreadCount 
      get -d Catalina -b name=http-80,type=ThreadPool currentThreadsBusy
      quit

      Output:


      Welcome to JMX terminal. Type "help" for available commands.
      #Connection to localhost:3993 is opened
      #mbean = Catalina:port=80,type=Connector:
      acceptCount = 100;

      #mbean = Catalina:name=http-80,type=ThreadPool:
      maxThreads = 40;

      #mbean = Catalina:name=http-80,type=ThreadPool:
      currentThreadCount = 18;

      #mbean = Catalina:name=http-80,type=ThreadPool:
      currentThreadsBusy = 5;

      #bye

note #1 : if domain si not set, you must give it  to get command as parameter:
get -d Catalina -b ${bean_name} ${attribute_name}

Wrapping in Python

The problem with jmxterm si that its startup time is really long - even several seconds. Also connecting to process can take second or two. The reponsivness is good so if you need it for periodic monitoring, you would like to start it and connect and then fire requests when you need the data. After trying some approaches I settled on Python pexpect for this (script reduced to minimum):
import time
import pexpect

connection = "localhost:3993"
connection_timeout = 2

jmxterm = pexpect.spawn("java -jar jmxterm-1.0-alpha-4-uber.jar")
jmxterm.expect_exact("$>") # got prompt, we can continue
jmxterm.sendline("open " + connection)
jmxterm.expect_exact("#Connection to "+connection+" is opened", connection_timeout)

jmxterm.sendline("get -d Catalina -b name=http-80,type=ThreadPool currentThreadCount")

response_lines = []
response_lines.append(jmxterm.readline())
response_lines.append(jmxterm.readline())
response_lines.append(jmxterm.readline())
response_lines.append(jmxterm.readline())

result = response_lines[3].replace(";"," ").strip().split(" ")
del result[1]
name,value = result

print "["+time.ctime()+"]", name, "=", value
jmxterm.sendline("quit")

3 comments:

  1. Thank you for this, it's very helpful.
    I was trying to do this with subprocess ... pexpect was what i needed !!

    ReplyDelete
  2. Hello,

    very interesting and powerfull. Is it possible to daemonize in some way this python handler and parse queries to it?

    ReplyDelete
  3. I don't see why it should not be possible. Look at the Twisted (twistedmatrix.com) framework for example, they have even a document about writing servers using Twisted API.

    ReplyDelete