Learning Terracotta: An Example

Whilst Terracotta does not require you to develop code specifically with distribution in mind, in the long run it can help with overall performance if the data shared between the master and workers is minimised. The more autonomously the workers can perform their operations, the less the Terracotta server needs to manage across the multiple JVM’s.

Developing the solution with this in mind involved some trial and error for us. To help explain some of the hurdles encountered, we’ll present a simplified example.

Our application processes multiple files containing records composed of comma-separated values (CSV). Furthermore, each CSV record contains a field called a Maximum Daily Usage (MDU). For the sake of simplicity, lets just say that, amongst other things, our task was to report the maximum MDU found for each file (the actual processing is far more complicated than this).

Initially, as per Jonas’ blog entry, we created a WorkUnit that implemented the Work interface from the CommonJ WorkManager specification. This WorkUnit was responsible for processing a given file to find the maximum MDU. The code looked something like this:

public class WorkUnit implements Work
{
    private String filePath;
    private Reader reader;    

    public WorkUnit(String aFilePath)
    {
        filePath = aFilePath;
    }

    public void run()
    {
        private String maxMDU = "";
        setReader(new BufferedReader(new FileReader(filePath)));   
        String record = reader.readLine();
        while (record != null)
        {
            String[] fields = record.split(",");
            String mdu = fields[staging:4];
            if (mdu.compareTo(maxMDU) > 0)
            {
                setMaxMDU(mdu);
            }
            record = reader.readLine();
        }
        System.out.println(“maximum MDU = “ + maxMDU);

        doStuffWithReader();
        doMoreStuffWithReader();
        doEvenMoreStuffWithReader();
    }

    private void setReader(Reader reader)
    {
        this.reader = reader;
    }
    …
}

For each file in a given directory, the Master created a WorkUnit, providing it with location of the file it was to process. The WorkUnit was then wrapped in a WorkItem – which included a status flag - and ‘scheduled’ by placing it on a shared queue. Our Worker implementations then picked up the WorkItem, extracted our WorkUnit and called its run() method, which in turn would process it’s file and report the maximum MDU encountered. It would then go onto do a bunch of other things with the file.

The key thing to note is that we’ve chosen to store the Reader as an instance variable - it’s actually being used by a number of methods and it would be impractical to use a local variable and pass it around everywhere.

To share the queue amongst workers, we configured Terracotta’s tc-config.xml as follows:

<application>
  <dso>
    <roots>
      <root>
        <field-name>
          com.shinetech.mrs.batch.core.queue.SingleWorkQueue.m_workQueue
        </field-name>
      </root>
    </roots>
    <locks>
      <autolock>
        <method-expression>* *..*.*(..)</method-expression>
        <lock-level>write</lock-level>
      </autolock>
    </locks>
    <instrumented-classes>
      <include>
        <class-expression>
          com.shinetech.mrs.batch.core.queue..*
        </class-expression></include>
      <include>
        <class-expression>
          com.shinetech.mrs.batch.input.workunit..*
        </class-expression>
      </include>
    </instrumented-classes>
  </dso>
</application>

You don’t need to understand this in too much detail other than that it nominates the fields to be shared – in this case our queue - and specifies which classes should be instrumented to safely share it.

Our first run using this WorkUnit failed - when the WorkUnit tried to set the value of the reader variable an UnlockedSharedObjectException was thrown by Terracotta with the message “Attempt to access a shared object outside the scope of a shared lock”. Essentially Terracotta was telling us that we had tried to update an attribute of an object that was shared between the Master and the Worker, but we had not told Terracotta that it needs to lock (or synchronize) the attribute.

The key problem here is that even if we don’t initialise the instance variable until we run the worker, Terracotta regards it as shared by the master and the worker because the variable belongs to an object on a queue that is shared. (Incidentally, the Terracotta exception handling is excellent; if you try to do something that you can’t, a Terracotta exception will tell you both what you did wrong and what you should do to fix it.)

At this stage there were a few different approaches we could have taken. One option was to have synchronized setReader() in the WorkUnit class. Terracotta would have then locked access to the reader thanks to the autolock section in tc-config.xml. Alternatively, rather than changing the code, we could have simply added a named-lock to the locks section of tc-config.xml, which in turn would have told Terracotta to effectively synchronize access to the reader across the JVM cluster:

<named-lock>
  <lock-name>WorkUnitSetterLock</lock-name>
    <method-expression>
      * com.shinetech.mrs.batch.dataholder..*.set*(..)
    </method-expression>
  <lock-level>write</lock-level>
</named-lock>

Ultimately, however, neither of these approaches was what we needed. For in a more realistic and complex example, the WorkUnit may have many instance variables that are really only relevant when it is being run by the Worker. The Master does not need to know anything about them, and they exist really only for the duration of the run() method. From a performance perspective, we didn’t want Terracotta to have to synchronize access to these WorkUnit attributes if the Master will never access them.

What we really wanted was the ability for the WorkUnit to be instantiated not by the master, but instead at the time that the Worker picked up the WorkItem off the queue. To accomplish this we introduced a WorkUnitGenerator:

public class WorkUnitGenerator implements Work
{
    private String filePath;
    
    public WorkUnitGenerator(String aFilePath)
    {
        filePath = aFilePath;
    }

    public void run()
    {
        WorkUnit workUnit = new WorkUnit(filePath);
        workUnit.run();
    }
}

Now, the Master creates a WorkUnitGenerator, providing it with the location of the file to process. The WorkUnitGenerator is wrapped in a WorkItem and scheduled. The Worker implementation picks up the WorkItem, extracts the WorkUnitGenerator and calls its run() method. The run() method instantiates a new WorkUnit and delegates the file processing to the WorkUnits run() method. So we now have a situation where the WorkUnit, which in our case should be independent of the Master, is indeed independent, and Terracotta doesn’t need to perform any unnecessary synchronization across JVM’s.

The code samples above outline just one approach to the Master / Worker pattern. There are probably other ways it could be done, and down the track after more experience with Terracotta, we may well find a better implementation. The main point is that whilst Terracotta and the Master / Worker pattern give you the ability to distribute work across many machines, you have to be careful about what you want to share and what you don’t.