Random Developments: September 2009

In my last post I had almost everything working for my first POC interface that transformed a fixed width file to an XML message. What I was missing was putting that XML on an IBM MQ queue.

This I thought would be a snap! But I had a little trouble until I finally got it to work. My solo attempt to finish off this part of the POC was met with the usual cast of characters: "class not found" errors, version issues and general noob mistakes.

IBM MQ JMS Support

After consulting my fearless Sales Solutions Architect, Puneet, he was able to untangle my MQ and JMS configuration. I am still a little fuzzy on how the JMS endpoint is aware of the MQ connection factory when there is no explicit linkage between the two. In any event, here is the bits that allow writing my XML message to an IBM MQ queue:

<spring:bean id="jmsWmqConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<spring:property name="hostName" value="localhost"/>
<spring:property name="port" value="1414"/>
<spring:property name="queueManager" value="DEVQMGR"/>
<spring:property name="transportType" value="1"/>
<spring:property name="channel" value="SYSTEM.DEF.SVRCONN"/>
</spring:bean>

<jms:websphere-connector name="JmsConnector" connectionFactory-ref="jmsWmqConnectionFactory"
disableTemporaryReplyToDestinations="true"/>

The jmsWmqConnectionFactory establishes the JMS connection factory. Later I would like to use a JNDI configuration and lookup to get these property values. The obvious reason is to allow environment specific configurations that can be model and replaced at build time. For now though I am just hard coding the connection properties to my local development queue manager. I am using IBM MQ v6.0.2.6.

The next piece in the JMS configuration is the JmsConnector. The JmsConnector references the previously defined connection factory jmsWmqConnectionFactory. And now, the final bit of magic, the JMS outbound endpoint:

<jms:outbound-endpoint queue="QR.12.WMS.IN.PROD_DIM_DL" synchronous="true"/>

I placed this outbound endpoint after my previously defined inbound endpoint and transformers.

Running my now complete cubiscan POC Mule configuration, the following output is generated in the console:

* Server started: 9/30/09 11:31 AM *
* Server ID: 6eb443e8-adef-11de-88f5-41bec82dcb83 *
* JDK: 1.6.0_14 (mixed mode, sharing) *
* OS encoding: Cp1252, Mule encoding: UTF-8 *
* OS: Windows XP - Service Pack 3 (5.1, x86) *
* Host: DEVBXA1-D620 (10.8.25.176) *
* *
* Agents Running: None *
**********************************************************************
[09-30 11:31:31] INFO FileMessageReceiver [fileConnector.receiver.1]: Lock obtained on file: D:\workspace\CubiscanPOC\data\in\Fr152335.csb
[09-30 11:31:31] INFO JmsMessageDispatcher [fileConnector.receiver.1]: Connected: endpoint.outbound.jms://QR.12.WMS.IN.PROD_DIM_DL

I can now browse my queue manager and see the XML message on the QR.12.WMS.IN.PROD_DIM_DL queue. Success!!!

Things I Learned

I had to add a couple of jars that were not part of my initial Mule build path:

mule-jms-provider-1.2.jar
mule-transport-jms-2.2.1.jar

I had to download these from the Mulesoft website and add to my java build path in Eclipse.

I also had to add an MQ jar:

IBM_LIB/com.ibm.mqjms.jar

Where IBM_LIB is where your local IBM jars reside. Mine was located at C:/Program Files/IBM/WebSphere MQ/Java/lib

I kind of got lost trying to follow some example in other online posts and books. But with the variety of JMS providers out there and differences between version 1.x and 2.x of Mule, my original JMS configuration was broken. But no worries, it all got sorted as I said with Puneet's help. Thanks Puneet.

Here is the final Mule configuration:

<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesource.org/schema/mule/core/2.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:stdio="http://www.mulesource.org/schema/mule/stdio/2.2"
xmlns:file="http://www.mulesource.org/schema/mule/file/2.2"
xmlns:xm="http://www.mulesource.org/schema/mule/xml/2.2"
xmlns:vm="http://www.mulesource.org/schema/mule/vm/2.2"
xmlns:jms="http://www.mulesource.org/schema/mule/jms/2.2"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.mulesource.org/schema/mule/core/2.2 http://www.mulesource.org/schema/mule/core/2.2/mule.xsd
http://www.mulesource.org/schema/mule/stdio/2.2 http://www.mulesource.org/schema/mule/stdio/2.2/mule-stdio.xsd
http://www.mulesource.org/schema/mule/file/2.2 http://www.mulesource.org/schema/mule/file/2.2/mule-file.xsd
http://www.mulesource.org/schema/mule/vm/2.2 http://www.mulesource.org/schema/mule/vm/2.2/mule-vm.xsd
http://www.mulesource.org/schema/mule/xml/2.2 http://www.mulesource.org/schema/mule/xml/2.2/mule-xml.xsd
http://www.mulesource.org/schema/mule/jms/2.2 http://www.mulesource.org/schema/mule/jms/2.2/mule-jms.xsd">
<description> </description>

<file:connector name="fileConnector" autoDelete="false" streaming="false"
moveToDirectory="data/archive" fileAge="10000" pollingFrequency="5000"
moveToPattern="#[ORIGINALNAME]_#[SYSTIME]"/>

<custom-transformer name="FixedWidthToMapsTransformer"
class="com.mulesource.mule.transport.jdbc.transformers.FixedWidthToMapsTransformer">
<spring:property name="mappingFile" value="map.cubiscan.xml"/>
</custom-transformer>

<custom-transformer name="MapsToXMLTransformer"
class="com.mulesource.mule.transport.jdbc.transformers.MapsToXMLTransformer"> </custom-transformer>

<xm:xslt-transformer name="Xslt" xsl-file="xsl/cubiscanMap2WMS.xslt"/>

<spring:bean id="jmsWmqConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<spring:property name="hostName" value="localhost"/>
<spring:property name="port" value="1414"/>
<spring:property name="queueManager" value="DEVQMGR"/>
<spring:property name="transportType" value="1"/>
<spring:property name="channel" value="SYSTEM.DEF.SVRCONN"/>
</spring:bean>

<jms:websphere-connector name="JmsConnector" connectionFactory-ref="jmsWmqConnectionFactory"
disableTemporaryReplyToDestinations="true"/>





<model name="cubiscanPOC">

<service name="Cubiscan2XML">
<inbound>
<file:inbound-endpoint path="data/in"
transformer-refs="FixedWidthToMapsTransformer MapsToXMLTransformer Xslt"
synchronous="true"/>
</inbound>

<outbound>

<pass-through-router>
<jms:outbound-endpoint queue="QR.12.WMS.IN.PROD_DIM_DL" synchronous="true"/>
</pass-through-router>
</outbound>
</service>

</model>

</mule>

Next I want to add simple JNDI lookup and move on to my next POC interface. My next POC interface will read from a database and generate a CSV file email attachment.

Two simple flat file interfaces were chosen for a proof of concept (POC) to compare Mulesoft's Mule and Apache's ServiceMix ESBs. The two interfaces chosen were: cubiscan to warehouse (Cubiscan2WMS) and order management system refund check to finance department (OMSRefundCheck2Finance).

These interfaces were chosen for two primary reasons: they are simple flat files and they involve common and unique transformations. Cubiscan2WMS involves a fixed width, flat file transformation to XML. It also is triggered by a file event/directory listener. File listeners are a common interface pattern at Gap Inc. Direct (GID). OMSRefundCheck2Finance reads it's input from a database table and generates email attachments. Reading from a database (as well writing) are common integration patterns at GID. Email generation is not a common integration pattern at GID but I was interested in discovering how Mule and ServiceMix address this requirement.

Cubiscan2WMS

The Cubiscan2WMS interface processes product item dimension data (width, height, length, and weight) to determine optimal packaging during order fulfillment. As product arrives at distribution centers (DCs), items are scanned by a third party application/device (CubiScan). Item dimensions are stored in a database item master table indexed by SKU.

OMSRefundCheck2Finance

The OMSRefundCheck2Finance interface process exception refunds due to customers from the Order Management System (OMS) to the finance department as an email with Excel attachments. Normal refunds are credited back to customers credit cards but certain refunds that are exceptions are processed with OMSRefundCheeck2Finance. The interface runs once daily with emails being sent to a finance department distribution email list. The finance department manually create checks from the information in the attached spreadsheets and sends to customers via snail mail.

Mule POC

This past Thursday (September 10th), I met with Puneet Gupta (yet another Gupta!) from MuleSoft. Puneet is a Sales Solution Architect and was accompanied by Paul Davidian, Account Executive for our 1PM meeting at Gap. Paul and I exchanged some pleasantries and he was off leaving me and Puneet to gt down to business.

I had spent the week prior gathering all the pertinent supporting documentation to help accelerate the POC working session. I had sent Puneet architecture diagrams, input, output samples and mapping documents for the POC interfaces. I also configured my development environment by installing the latest JDK, Eclipse and Mule.

Below was my configuration:

java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) Client VM (build 14.0-b16, mixed mode, sharing)

Eclipse Ganymede Version: 3.4.2

Mule Community Edition 2.2

*******************************************************************************
Mule ESB and Integration Platform
Version: 2.2.0
MuleSource, Inc. http://www.mulesource.com
*******************************************************************************

I also installed the Eclipse Mule IDE plugin version 1.0.0.v200903220319.

Puneet and I promptly got busy knowing we only had 2 hours scheduled for today's session. We chose to work on the Cubiscan2WMS interface first. I create a new Mule project in Eclipse with the Mule IDE plugin (File->New->Other->Mule->Mule Project). The new project wizard allows using an existing sample project to be used as a starting place. Puneet suggested using the "hello" sample.

Config.xml

The heart of the Mule project is the configuration file in the workspace /conf directory. The first thing I did was rename the config file and name references within to something more appropriate.

The config file is broken down into several sections:

* Schema references
* Transformers and connectors
* Model
* Services
* Endpoints

Knowing that our input endpoint was a file poller, we added a "file connector" in Mule speak. I added the file schema reference and the following snippet to the confg file:

<file:connector name="fileConnector" autoDelete="false"
streaming="false" moveToDirectory="data/archive" fileAge="10000"
pollingFrequency="5000" moveToPattern="#[ORIGINALNAME]_#[SYSTIME]" />

The properties are fairly straight forward but full details can be found on the Mule site:

http://www.mulesoft.org/display/MULE2USER/File+Transport

The properties I wanted to call out are autoDelete and streaming. I had set these both to false because I specified a moveToDirectory and did not want the file deleted until the file was moved to an archive location and I did not need streaming.

My model (cubiscanPOC) had one service called Cubiscan2XML. This service contained two endpoints: inbound and outbound.

<model name="cubiscanPOC">

<service name="Cubiscan2XML">
<inbound>
<file:inbound-endpoint path="data/in"
transformer-refs="FixedWidthToMapsTransformer MapsToXMLTransformer Xslt"
synchronous="true" />
</inbound>

<outbound>
<pass-through-router>
<stdio:outbound-endpoint system="OUT" />
</pass-through-router>
</outbound>
</service>

</model>

The inbound endpoint references is using the file connector described earlier and some transformers: FixedWidthToMapsTransformer MapsToXMLTransformer Xslt. For the file connector, I defined the input path to "data/in" which is in my Mule project workspace.

At first I tested without any transformers so the file contents got dumped to the output endpoint which is just a pass through router to stdio. This makes iterative testing easy.

Next I added each transformer, verifying results on the console from the output endpoint. The transformers appear in the Mule config file as:

<custom-transformer name="FixedWidthToMapsTransformer"
class="com.mulesource.mule.transport.jdbc.transformers.FixedWidthToMapsTransformer">
<spring:property name="mappingFile" value="map.cubiscan.xml" />
</custom-transformer>

<custom-transformer name="MapsToXMLTransformer"
class="com.mulesource.mule.transport.jdbc.transformers.MapsToXMLTransformer">
</custom-transformer>

<xm:xslt-transformer name="Xslt" xsl-file="xsl/cubiscanMap2WMS.xslt" />

FixedWidthToMapsTransformer

This transformer came out of the box from Mule. It converts a flat file to a name/value based stream using the map file to pull the fixed width fields to name, value pairs.

Here is what the definition of the map file looks like in map.cubiscan.xml:

<?xml version="1.0"?>

<!DOCTYPE PZMAP SYSTEM "flatpack.dtd" >

<PZMAP>
<COLUMN name="ITEMUPC" length="20" />
<COLUMN name="ITEMLENGTH" length="20" />
<COLUMN name="ITEMWIDTH" length="20" />
<COLUMN name="ITEMHEIGHT" length="20" />
<COLUMN name="ITEMDIMUOM" length="2" />
<COLUMN name="ITEMWEIGHT" length="20" />
<COLUMN name="ITEMWEIGHTUOM" length="2" />
<COLUMN name="ITEMVOLUME" length="20" />
<COLUMN name="ITEMVOLUMEUOM" length="2" />
<COLUMN name="ITEMDIMWEIGHT" length="20" />
<COLUMN name="FACTOR" length="11" />
<COLUMN name="SITEID" length="4" />
<COLUMN name="DC" length="2" />
<COLUMN name="TIMESTAMP" length="22" />
<COLUMN name="FILLER" length="122" />

</PZMAP>

And here is the output from the FixedWidthToMapsTransformer:

[{ITEMVOLUMEUOM=in, ITEMWIDTH=000000000000000012.5, ITEMWEIGHT=0000000000000001.300, DC=, ITEMHEIGHT=000000000000000001.1, FACTOR=194, ITEMLENGTH=000000000000000014.1, ITEMWEIGHTUOM=lb, SITEID=OFC, FILLER=, ITEMVOLUME=0000000000000193.875, TIMESTAMP=3:23:35 PM 6/19/2009, ITEMDIMWEIGHT=0000000000000001.000, ITEMUPC=161020010193, ITEMDIMUOM=in}]

MapsToXMLTransformer

The next transformer in the chain of transformers converts the map name, value output above and produces an XML equivalent.

After MapsToXMLTransformer, here is the output:

<?xml version="1.0" encoding="UTF-8"?>
<table>
<record>
<field name="ITEMVOLUMEUOM" type="java.lang.String">in</field>
<field name="ITEMWIDTH" type="java.lang.String">000000000000000012.5</field>
<field name="ITEMWEIGHT" type="java.lang.String">0000000000000001.300</field>
<field name="DC" type="java.lang.String"></field>
<field name="ITEMHEIGHT" type="java.lang.String">000000000000000001.1</field>
<field name="FACTOR" type="java.lang.String">194</field>
<field name="ITEMLENGTH" type="java.lang.String">000000000000000014.1</field>
<field name="ITEMWEIGHTUOM" type="java.lang.String">lb</field>
<field name="SITEID" type="java.lang.String">OFC</field>
<field name="FILLER" type="java.lang.String"></field>
<field name="ITEMVOLUME" type="java.lang.String">0000000000000193.875 </field>
<field name="TIMESTAMP" type="java.lang.String">3:23:35 PM 6/19/2009</field>
<field name="ITEMDIMWEIGHT" type="java.lang.String">0000000000000001.000</field>
<field name="ITEMUPC" type="java.lang.String">161020010193</field>
<field name="ITEMDIMUOM" type="java.lang.String">in</field>
</record>
</table>

Xslt

My time with Puneet ran out - our 2 hour block of time was at an end before we could finish the final transformation with xsl. But dealing with Mule was mostly intuitive and I felt confident I could finish it off. I thanked Puneet, he rushed off to his next engagement.

The next few days I squeezed in slivers of time to complete what i started with Puneet. I created an XSLT file and pointed to it in the transformer definition:

<xm:xslt-transformer name="Xslt" xsl-file="xsl/cubiscanMap2WMS.xslt" />

Next, reference the xslt transformer in the file endpoint transformer-refs list:

<file:inbound-endpoint path="data/in"
transformer-refs="FixedWidthToMapsTransformer MapsToXMLTransformer Xslt"
synchronous="true" />

After Xslt, we get our final XML output message ready to be consumed by our destination system, WMS:

<?xml version="1.0" encoding="UTF-8"?>
<Item OrganizationCode="GAPINC">
<ItemAliasList>
<ItemAlias AliasName="Vendor UPC Code" AliasValue="161020010193"/>
</ItemAliasList>
<PrimaryInformation UnitWidth="000000000000000012.5" UnitWeight="0000000000000001.300"
UnitHeight="000000000000000001.1"
UnitLength="000000000000000014.1"
UnitWeightUOM="lb"
UnitHeightUOM="in"
UnitLengthUOM="in"
UnitWidthUOM="in"/>
<InputXML>
<Item OrganizationCode="GAPINC">
<ItemAliasList>
<ItemAlias AliasName="Vendor UPC Code" AliasValue="161020010193"/>
</ItemAliasList>
<PrimaryInformation UnitWidth="000000000000000012.5" UnitWeight="0000000000000001.300"
UnitHeight="000000000000000001.1"
UnitLength="000000000000000014.1"
UnitWeightUOM="lb"
UnitHeightUOM="in"
UnitLengthUOM="in"
UnitWidthUOM="in"/>
</Item>
</InputXML>
</Item>

Wrap Up

So concluded the first day of Mule POC. All in all, i was fairly impressed how easy it was to go from nothing to having something working without a lot of heavy lifting. Obviously there will be more work to 'productionalize' this solution including how to integrate this with our development environment and build pipeline.

Next post, I will attempt to send the resulting WMS message to an IBM MQ queue.

Here is the entire Mule cubiscan-config.xml:

Random Developments

Tuesday, September 29, 2009

Mule POC Part II

Tuesday, September 15, 2009

Mule POC Results

Followers

Blog Archive

About Me