GlassFish Production Tuning

So there you are: You’ve started working on your state-of-the-art Java EE application, GlassFish is your application server of choice and it’s time to set up automatic deployment of the application to your as-equivalent-to-production-as-possible staging environment for periodical stress testing of nightly builds. But should you do your testing on GlassFish configured as it comes out of the box?

Probably not. A key to successful stress testing is to tune your GlassFish installation with the same configuration as you would in your production environment. Without an identical configuration, the value of the stress tests will drop dramatically; even if your application passes the tests with flying colors and ticker-tape parades on your staging platform, you might still crash and burn miserably in production.

This small guide aims to show you at least some of the hot spots you need to know of when tuning your GlassFish installation for a production environment. We’ll focus on the JVM and GlassFish, and let others take care of configuring the rest of the system, like optimizing the JDBC driver configuration. I did this tuning on GlassFish 3.0.1, but it’s very likely that everything here also applies to the recently released 3.1 version.

Throughout this entire guide I assume that you have some basic experience, both with Java and GlassFish. When I use ellipsis (…), this does not mean that you should look for or replace the current configuration with three dots, it means that there is something in the configuration that I’ve removed for clarity.

JVM Tuning

Let’s start with a little JVM tuning. This is done in domain.xml, located in the %GLASSFISH_HOME%/glassfish/domains/<DOMAIN_NAME>/config directory. Make sure that your GlassFish instance is not running when you modify the domain.xml file. When a GlassFish domain is shut down, it saves the current domain configuration to this file and every edit you’ve made will be lost.

The JVM parameters are found inside the java-config element in the file:

<java-config classpath-suffix="" system-classpath="" debug-options="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009">
        <jvm-options>-XX:MaxPermSize=192m</jvm-options>
        <jvm-options>-client</jvm-options>
        ...
</java-config>

The first thing you want to do is to allocate more RAM to the JVM by modifying the value of the -Xmx flag. How much depends on the size and complexity of your enterprise application, but I’ve yet to see one that behaves well over time with less than 1GB of RAM to play with, so let’s start with that. In addition to giving the JVM a maximum of 1GB of RAM, we also want make sure it allocates all of that RAM on startup so it doesn’t have to waste time doing that later. This is done with the -Xms flag.

<jvm-options>-Xmx1024m</jvm-options>
<jvm-options>-Xms1024m</jvm-options>

Next we want to replace the -client flag with -server. This will tell the JVM to continuously analyze the code it executes and modify it for optimal performance. To measure the performance gain when using -server instead of -client is hard because it will take some time before the server is happy with the code it executes. The important part is that this all happens under the hood and that you can get a performance boost for free.

<jvm-options>-server</jvm-options>

Note that we’re not doing any explicit garbage collection tuning. The reason for this is that GC tuning is rarely necessary when running on a recent JVM version.

GlassFish Tuning

Now for some reconfiguration of the application server itself. First, we want to turn off auto-deploy and dynamic application reloading. Both are great for development, but can have nasty side effects in terms of performance. Turn off both by modifying domain.xml:

<das-config autodeploy-enabled="false" dynamic-reload-enabled="false" ... />

Note that in Glassfish 3.1, the das-config entry is not available by default in the domain.xml file. Instead, you can change these settings from the Glassfish administration console. The latter also applies to Glassfish 3.0, but you can also change it in the domain.xml file if you prefer that.

Some settings are easier to change using the GlassFish administration console rather than poking around in the configuration files themselves. One example is acceptor threads, which should be set to a number equal the number of CPU cores in your server. So, if you have 2 CPUs with 4 cores each, the value should be 8. Log in to the administration console at http://<server-ip>:4848/ and go to

Configuration -> Network Config -> Transports -> tcp -> Acceptor threads

It’s also recommended to cache your static resources, like CSS files and images. By default, GlassFish does not tell the client to cache static resources and you can get a nice performance gain by turning this on if you have a lot of them. In the administration console, go to

Configuration -> Network Config -> Protocols -> http-listener-1 -> File Cache -> Enabled

If you also use secure connections (HTTPS), make sure you turn on the file cache on http-listener-2 as well.

If your server only has one network interface card (NIC) installed, replace all 0.0.0.0 values in your network listeners configuration with the IP address of the server. By doing this, the server will make one less system call per connection. Configure all your listeners in

Configuration -> Network Config -> Network Listeners -> <listener_name> -> Address

Let’s move on to another file, default-web.xml. This file is also located in the %GLASSFISH_HOME%/glassfish/domains/<DOMAIN_NAME>/config directory.

Here we will make some subtle, but important changes to the configuration of the JspServlet. First, we want to configure the servlet not to check the JSP files for changes on every request. Second, we want all String values to be declared as static char arrays. One reason for this is that the array has less memory overhead than String.

<servlet>
    <servlet-name>jsp</servlet-name>
    <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
    <init-param>
        <param-name>development</param-name>
        <param-value>false</param-value>
    </init-param>
    <init-param>
        <param-name>genStrAsCharArray</param-name>
        <param-value>true</param-value>
    </init-param>
    ...
</servlet>

As a last friendly tip; remember to turn off as much logging as possible. Ideally, your application should not log anything except for errors in the production environment. Having info logging on might be tempting, but this will most likely result in a massive CPU overhead under heavy load and logging might be what breaks the virtual legs of your application. This goes for both GlassFish’ own logging (which can be configured in the logging.properties file in the same directory as all the other files we’ve been looking at) and the logging done by your application.

In this guide we have looked at the most essential GlassFish tuning tips, but we’ve not touched the database pool configuration. These settings depend very much on how database intense your enterprise application is. It also depends on your database, what load can it handle without crumbling? Database pool tuning is an art in itself, and unfortunately something I don’t have a lot of experience with.

Sources:

This guide is also available on BEKK Open.


Feedback

Do you have any thoughts you want to share? A question, maybe? Or is something in this post just plainly wrong? Then please send an e-mail to vegard at vegard dot net with your input. You can also use any of the other points of contact listed on the About page.

Caution

It looks like you're using Google's Chrome browser, which records everything you do on the internet. Personally identifiable and sensitive information about you is then sold to the highest bidder, making you a part of surveillance capitalism.

The Contra Chrome comic explains why this is bad, and why you should use another browser.