Wednesday, November 28, 2012

Don’t let tx:annotation-driven get you


When we start developing a new application, we typically have a single data source that we define and all of our business logic revolves around getting data back and forth the data source. As the application matures, we often run into scenarios where we need to depend on jars given my others or we give out our jars to others for them to use. With Spring involved, the context also becomes part of the contract between the two sub-systems. Here is where tx:annotation-driven can get you. Typically, we define a PlatformTransactionManager of org.springframework.jdbc.datasource.DataSourceTransactionManager and bind it to the datasource that it should manage.
    <bean id="testerDataSource" class="org.apache.commons.dbcp.BasicDataSource">
        <property name="driverClassName" value="net.sourceforge.jtds.jdbc.Driver" />
        <property name="url" value="${tester.datasource.url}" />
        <property name="username" value="${tester.datasource.username}" />
        <property name="password" value="${tester.datasource.password}" />
    </bean>
    <bean id="testerTransactionManager"
        class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
        <property name="dataSource" ref="testerDataSource" />
    </bean>
Next, to achieve declarative transactional functionality on specific methods in our service, we can use tx:advice in conjunction with aop:pointcut to indicate the advice that will act on the methods matching the pointcut. However, I prefer defining annotations on the service method itself to effect this declaratively. The upside of this, in my opinion, is that the reader of the code is clearly made aware of what are the transactional boundaries the service method is going to honor.
    @Transactional
    public void getReferenceData() {
        referenceDataDAO.getReferenceData();
    }
From Spring’s manual:
Declaring transaction semantics directly in the Java source code puts the declarations much closer to the affected code. There is not much danger of undue coupling, because code that is meant to be used transactionally is almost always deployed that way anyway.
To get @Transactional to work its magic, i.e. create a proxy which does the heavy lifting of beginning the transaction, delegating the call to the actual class, and either committing or rolling back the transaction based on the result of the call, we need to inform Spring to scan all beans that have this annotation on it. We inform Spring using the “tx:annotation-driven” directive. It is a namespace handler which scans all beans in the Spring container to proxy all methods that have been annotated with @Transactional. The @Transactional can also be placed at the class level, when it is applicable to all methods in the class. So this namespace handler will by default look for a bean named ‘transactionManager’ of class PlatformTransactionManager. If your transaction manager bean happens to be named any differently, you have to state it explicitly. Another note of caution from Spring:
Method visibility and @Transactional
When using proxies, you should apply the @Transactionalannotation only to methods with public visibility. If you do annotate protected, private or package-visible methods with the@Transactional annotation, no error is raised, but the annotated method does not exhibit the configured transactional settings. Consider the use of AspectJ (using mode=”aspectj”) if you need to annotate non-public methods.
Using the “proxy-target-classes” clause, we are able to choose whether we create interface-based JDK proxies or CGLIB based proxies. The former will only work if the class you are proxying is implementing an interface whereas CGLIB-based ones work with either one. OK, so far so good, how is that tx:annotation-driven is going to get me? Just when it is all so comfortable, there is another data source that now comes in the context, either because you imported someone else’s jar or because you started talking to a new data source, tx:annotation-driven will try to hijack any @Transactional annotations intended for those data sources as well and ask the existing transaction manager to try and manage it. All that the transaction manager does is hold a connection from the data source it had been configured with in a threadlocal. When trying to manage a hijacked data source the threadlocal that it has kept with it is non-consequential in rolling back the transaction if required. The worst part is that there is no error that is thrown, just that the effect will not happen.
On a related note, because the connection is held in a threadlocal, if you are spawning threads or delegating control to other threads from a  method that is annotated with @Transactional, all bets are off and you might end up with infinite waits or deadlocks.
So, what is the recommended solution? Spring suggests that we qualify your transaction managers and rather than giving a bare @Transactional, specify the transaction manager (via qualifier) that you intend to use. In addition to this, one can also choose to avoid specifying the transaction manager when specifying the tx:annotation-driven which will force every user of @Transactional to specify the transaction manager it wishes to use (otherwise the container throws an exception saying that it didn’t find a bean with the name ‘transactionManager’). While this is more work, I think it is worth the clarity it provides and makes it explicit for users of the service method in other contexts.
References:
1. http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/transaction.html#transaction-declarative
2. Test bed available at https://github.com/kilokahn/spring-testers/tree/master/spring-tx-annotation-tester

Wednesday, November 21, 2012

My first one... on blogspot :)

I thought I should have a place where I can store all of the random thoughts that I come across everyday. Initially, I was maintaining a blog at my workplace, but for ready access I am thinking I will link it to my gmail account. It would be interesting to see if I can export blogs from there (apparently it is a wordpress based platform) in here. Update: I was able to import most blogs - yay! Have to blur out all the firm specific details in there before publishing them which reminds me that I need to install something like GIMP which can give me a blur tool rather than my present approach of using erase from mspaint! :)

Final note of caution: This is meant more as a place where I vent out my thoughts and rants on where and how (mostly technology) issues can be bettered.

Thursday, November 1, 2012

Spring PropertyPlaceholderConfigurer Tip

With the increased scrutiny on preferring to keep passwords in files and not as part of the source code, our reliance on PropertyPlaceholderConfigurer (PPC) has greatly increased. An important property of a PPC is the ignoreUnresolvablePlaceholders which is by default set to false. Setting this to false means that if any placeholders being passed to it were not resolved by it, it throws an exception and the entire context load fails. With the existence of multiple PPC's in a spring context (many of our own plus a few from all the transitive contexts we load), it helps to understand the interplay between them. Ideally every modules' that defines a PPC should necessarily have it's ignoreUnresolvablePlaceholders to true - and most do it as it will cause a context failure on loading. However, one may argue that the topmost context may need to know if all properties from all contexts were correctly resolved. By setting the ignoreUnresolvablePlaceholders to true, you may end up loading the context but having placeholders unresolved. The order in which the different PPC's are consulted is determined by the order property. However, if it is not defined, it defaults to Integer.MAX_VALUE. If order is not defined in any of the PPC's, the order is slightly non-deterministic but will mostly correspond to the order in which they are defined. The reason I say mostly is because Spring can choose to eagerly initialize beans to resolve circular dependencies. This is not limited to PPC's but also encompasses our vanilla beans referring other beans. This has been a topic that I believe we don't really pay a lot of attention to - but bites when you are least expecting. Anyway, there is a spring ticket out there to also have an "order" for vanilla beans as well. Coming back to ensuring that the context load is successful only when all placeholders have been able to be resolved - you can define an empty PPC that will act as the signaller for any unresolved placeholders. We will not define any locations for this PPC, and will have an order that is the highest among all - say Integer.MAX_VALUE with the default value of ignoreUnresolvablePlaceholders as false. Since this is the last guy that will get triggered, any unresolved placeholders will be flagged and context load stopped. Ofcourse this depends on the premise that the contexts (from different jars) that the topmost context loads will not have PPC's that are defined without order (defaulting to Integer.MAX_VALUE) and hence will not get loaded after the empty PPC in the topmost context.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

    <bean id="ppc1"
        class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <list>
                <value>file:///tmp/p1.properties</value>
            </list>
        </property>
        <property name="ignoreUnresolvablePlaceholders" value="true" />
        <property name="order" value="10000" />
    </bean>

    <bean id="ppc2"
        class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <list>
                <value>file:///tmp/p2.properties</value>
            </list>
        </property>
        <property name="ignoreUnresolvablePlaceholders" value="true" />
        <property name="order" value="10001" />
    </bean>

    <bean id="overallPPC"
        class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="order" value="20000" />
    </bean>

    <bean id="simpleBean" class="com.kilo.SimpleBean">
        <property name="property1" value="${com.kilo.property1}" />
        <property name="property2" value="${com.kilo.property2}" />
        <property name="property3" value="${com.kilo.property3}" />
        <property name="property4" value="${com.kilo.property4}" />
    </bean>

</beans>

So all you nice people out there giving out their jars with spring contexts to others, please ensure that your set an order of your PPC other than the default of Integer.MAX_VALUE. A good rule of thumb could be to use 10000-90000 for our own non-toplevel PPC.

References:

1. http://tarlogonjava.blogspot.in/2009/02/tips-regarding-springs.html
2. Sample setup available at https://github.com/kilokahn/spring-testers/tree/master/spring-ppc-tester


Monday, October 29, 2012

Classpath of a running java process spawned via maven exec:java

Ran into a peculiar situation where I was using the maven exec:java plugin to launch a java process. The plugin does the heavy lifting of creating the correct classpath comprising of its dependencies (much like what JUnit does). In this case, I suspected the java and maven to be conspiring against me by picking up a staler jar. In the case of Tomcat (or other containers) it is usually easy to look at WEB-INF/lib for the jars that will be used. But in this case, the command line didn't yield much information.

> ps -ef | grep produser| grep java
munshi     953 27405  0 01:21 pts/0    00:00:00 grep java
munshi   31771 31758 11 01:14 pts/0    00:00:50 /usr/local/java/jdk/bin/java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8765,suspend=n -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8219 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -classpath /tools/apache/maven/boot/plexus-classworlds-2.4.jar -Dclassworlds.conf=/tools/apache/maven/bin/m2.conf -Dmaven.home=/tools/apache/maven org.codehaus.plexus.classworlds.launcher.Launcher --settings /u/produser/.m2/settings.xml clean compile exec:java -Dexec.mainClass=com.kilo.DriverCLI -Dexec.args=SOMETHING 

The -classpath /tools/maven/boot/plexus-classworlds-2.4.jar didn't help much here. In JUnit, a whole file is created with the list of jars being used and it is very easy to inspect. There are intrusive ways to print the classpath for a application by either 1. setting -verbose:class when starting the process or by explicitly requesting in code with the system property "java.class.path". I fired up JVisualVM to see if it can help me examine the system property non-instrusively (since I really didn't want to restart the application for investigative reasons -- just yet). Again it showed the  /tools/maven/boot/plexus-classworlds-2.4.jar as the value. That's when I came across this alternate way where you use the lsof command to see which files are open. Since the classloader has to refer to the JAR from where the class is being loaded, it shows up in the list of open files - however not sure if this is a sureshot way to get things done - what if the JVM decides to close off the connection to the JAR as a way of reducing number of open file descriptors.

> lsof -p 31771 | less
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
java    31771 produser  cwd    DIR               0,70     4096 13085795 /u/produser/testproject (filer.kilo.com:/vol/proj1/proj-302)
java    31771 produser  rtd    DIR              253,0     4096        2 /
java    31771 produser    txt    REG              253,0     7630  1465254 /usr/local/java/jdk1.7.0.04/bin/java
java    31771 produser  mem    REG              253,0   156872  1441803 /lib64/ld-2.12.so
java    31771 produser  mem    REG              253,0  1922112  1441807 /lib64/libc-2.12.so
java    31771 produser  mem    REG              253,0    22536  1444029 /lib64/libdl-2.12.so
java    31771 produser  mem    REG              253,0   145720  1441859 /lib64/libpthread-2.12.so
java    31771 produser  mem    REG              253,0   598800  1444297 /lib64/libm-2.12.so
java    31771 produser  mem    REG              253,0    47064  1441866 /lib64/librt-2.12.so
java    31771 produser  mem    REG              253,0   113952  1444031 /lib64/libresolv-2.12.so
java    31771 produser  mem    REG              253,0   124624  1444032 /lib64/libselinux.so.1
java    31771 produser  mem    REG              253,0    17256  1444294 /lib64/libcom_err.so.2.1
java    31771 produser  mem    REG              253,0    12592  1444030 /lib64/libkeyutils.so.1.3
java    31771 produser    mem    REG              253,0   915104  1444295 /lib64/libkrb5.so.3.3
java    31771 produser  mem    REG              253,0    43304  1444292 /lib64/libkrb5support.so.0.1
java    31771 produser  mem    REG              253,0   181608  1444293 /lib64/libk5crypto.so.3.1
java    31771 produser  mem    REG              253,0   268944  1444296 /lib64/libgssapi_krb5.so.2.2
java    31771 produser  mem    REG              253,0     6398  1361340 /usr/local/java/jdk1.7.0.04/jre/lib/amd64/librmi.so
java    31771 produser  mem    REG              253,0  1023488  1361282 /usr/local/java/jdk1.7.0.04/jre/lib/ext/localedata.jar
java    31771 produser  mem    REG              253,0  2476995  1361287 /usr/local/java/jdk1.7.0.04/jre/lib/resources.jar
java    31771 produser  mem    REG              253,0     8934  1361283 /usr/local/java/jdk1.7.0.04/jre/lib/ext/dnsns.jar
java    31771 produser  mem    REG               0,71  1501575  1312261 /u/produser/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar (filer.kilo.com:/vol/home2/produser)

References:

1. http://thilinamb.wordpress.com/2009/07/01/analyse-the-classpath-of-a-running-java-program/

Tuesday, October 9, 2012

SQLServer Plan-Fu

Like google-fu, here is my MS SQL Server plan-fu :D

Finding out the cached plan handle for a query. You need some distinguishing text for the query - which I refer to as the MyMarker. Word of caution - this is computationally intensive so exercise discretion when running it - otherwise you might hear from your friendly neighborhood DBA-man :)
     SELECT cache_plan.plan_handle, cache_plan.objtype, cache_plan.size_in_bytes,
            cache_plan.cacheobjtype, cache_plan.usecounts, sql_text.text
       FROM sys.dm_exec_cached_plans as cache_plan WITH (NOLOCK)
OUTER APPLY sys.dm_exec_sql_text (cache_plan.plan_handle) as sql_text
      WHERE sql_text.text like N'%MyMarker%' AND cache_plan.cacheobjtype='Compiled Plan' AND cache_plan.objtype='Prepared'

To see the query plan using the plan_handle from above
SELECT query_plan
  FROM sys.dm_exec_query_plan (0x06003500F43BCC1940A17B9E010000000000000000000000)

To kick out the bad boy from the cache
DBCC FREEPROCCACHE (0x0600050093BEF30740014500030000000000000000000000)

To see the currently running query for a given spid
DBCC INPUTBUFFER (<spid>)

To see all the connections from a given login (to know the spid)
SELECT * FROM master..sysprocesses WHERE loginame='someuser'

Friday, October 5, 2012

Eclipse importing plugins and features from existing installation

Eclipse Juno released it's SR1 last week - so now seemed a right time to move away from our trusted Indigo SR2 to the Juno world. (Typically, we want to wait for an SR1 to ensure that all nagging issues with the new release are tackled with either fixes or workarounds are available from the community. Yes - this means that we are not on the bleeding edge, but caution in this case would be well-advised - after all we wouldn't want some nagging issue in the new release to set us back in productivity!) Update: Even the SR1 sucked!! Check out this if you don't believe me!

But what about the umpteen number of plugins that I had already installed with Indigo. You can either be lazy and wait for someone to tar an installation with most plugins required and then add the ones that are missing over again... or you can ask Eclipse to help you out. Starting from Indigo, Eclipse has a feature of importing the plugins and features from another installation. So you could download a vanilla copy of the JEE version from Eclipse's site and bring it up. Next, via File -> Import -> Installation -> From existing Installation, point it to where your previous Eclipse installation was present and presto, it will identify and import all compatible plugins. I have read about problems when plugins were incompatible, but luckily for me all plugins were compatible.

The code recommender and the JDT improvements are the two features of Juno that I am looking forward to the most!

References:

1. http://stackoverflow.com/a/11264964/874076

2. http://eclipsesource.com/blogs/2012/06/27/top-10-eclipse-juno-features/

Wednesday, October 3, 2012

Details of long running and long-forgotten processes

Sometimes we start off a new process and after quite sometime you wish to know details as to how the process was started. Things like current working directory, complete command line, environment variables used. This usually happens for the programs kicked off using a sudo-user and then you mostly quit the shell from where this was invoked. In my specific case, I was looking for the current working directory (to see where it will dump the core) and environment variables in use. In Linux, this is available in /proc/<pid>/ files.

I started browsing other entries and was amazed at the amount of useful information that is being transparently provided by Linux. Some of the notable ones are limits, statm and task. The reference link has details for other interesting details.

Here is the story:

I remember that I started the process on a particular host as the produser so log on there

> sudo -u produser ssh localhost

I remember that it is a java process - to get a handle to the process id.
> ps -ef | grep java | grep produser 
produser 8113  7983  0 02:14 pts/10   00:00:00 grep java
produser 21765 21750  0 Aug31 ?        01:46:45 /usr/local/java/jdk/bin/java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8765,suspend=n -Dcom.sun.management.jmxremote.port=8219 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -classpath /tools/apache/maven/boot/classworlds-1.1.jar -Dclassworlds.conf=/tools/apache/maven/bin/m2.conf -Dmaven.home=/tools/apache/maven org.codehaus.classworlds.Launcher "--settings" "/u/produser/.m2/settings.xml" "clean" "compile" "exec:java" "-Dexec.mainClass=com.kilo.DriverCLI" "-Dexec.args=SOMETHING"

Now check the current working directory:
> ls -al /proc/21765/cwd
lrwxrwxrwx 1 produser produser 0 Oct  1 08:04 /proc/21765/cwd -> /u/produser/testproject

Now for the environment settings:
> cat /proc/21765/environ
#All environment variables were listed - elided for privacy reasons

Other interesting info:
> cat /proc/21765/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             16341                16341                processes
Max open files            65536                65536                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       256591               256591               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

> cat /proc/21765/task/21771/status
Name: java
State: S (sleeping)
Tgid: 21765
Pid: 21771
PPid: 21750
TracerPid: 0
Uid: 5307 5307 5307 5307
Gid: 5307 5307 5307 5307
Utrace: 0
FDSize: 256
Groups: 5307 6135 6584
VmPeak: 11245832 kB
VmSize: 11243768 kB
VmLck:        0 kB
VmHWM:  3575560 kB
VmRSS:  2966532 kB
VmData: 11064932 kB
VmStk:       88 kB
VmExe:        4 kB
VmLib:    15324 kB
VmPTE:     6360 kB
VmSwap:        0 kB
Threads: 31
SigQ: 0/256591
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000004
SigIgn: 0000000000000000
SigCgt: 2000000181005ccf
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 8189
nonvoluntary_ctxt_switches: 933

References:

1. http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html