To me, the primary goal of an ESB is to provide a powerful, scalable, and easily extensible integration backbone.
“Integration” here means between end-points, applications, trading partners, etc… The basic premise is that you can throw a message at the ESB happens, and somehow that message finds its way to the correct endpoint(s), perhaps with a protocol hop or transformation occurring along the way, and maybe with a business process or two being invoked. In other words, throw a message at the ESB, and magic happens.
But, this is software, and there is no magic in software.
In order for us to process a message, we need to know what has to happen to it, which means we need some data (hereinafter: metadata) associated with the message that defines how a given message will move through the ESB.
Conceptually, you can think of this as an “envelope” for ESB messages. However, I am reluctant to use that term because it has different connotations for BizTalk developers. So, let’s just say “metadata”.
There are three ways I can think of (there are certainly more, but none I seriously considered) to potentially associate metadata with a message:
- Use an external metadata store
- Use context properties
- Use a multipart message
Let’s look at each option….
External Metadata store
Concept: Steps that happen are stored in some external store such as SQL Server, a controlling assembly, an XML document, etc.
I don’t like this and didn’t seriously consider it. You could do this, have a table somewhere that defines what to do with messages, but this just “feels” wrong. That information is related to a given message, and should travel with it as a message moves through the system. Consider: how would you version this? What about the latency you’d inject into the process as you look up what needs to happen? The more you think about what needs to be done, the more work and problems it starts sounding like. So, this was not a serious contender. This approach could work if the same set of properties were applied to all messages of a given type (think: “class variables”), but, in our ESB, we want a set of properties to be associated with each individual message (think: “instance variables”). Nice idea, wrong place to use it.
Using Context Properties
Concept: create our own message context properties that travel with a message through the ESB
This is how BizTalk does it. Data can be “promoted” from the message into message context properties or “demoted” from context properties into the message body. Context properties travel with a message for its lifetime, until it leaves BizTalk.
Using Multipart Messages
Concept: use BizTalk multipart-messages (a message constructed of multiple [maybe] instances of multiple [maybe] schemas, clear?)
This is much the same as above. This is perfectly viable, and would work, although promotion/demotion would be a little bit more complicated. Benefit is that you wouldn’t run into the 256 character limit that promoted properties are subject to. However, to me, a significant disadvantage would be that anyone applying maps in the ESB would need to map to/from a multipart message, meaning they would need to be aware of the multipart message format. With context properties, this would be transparent to a map developer, and maps could potentially be re-used outside the ESB.
What I chose
In the end, I opted to go with the context property approach, following the lead of the BizTalk team.
One of the facets of this project that is proving the most challenging is that because I am writing infrastructure components, EVERYTHING is by definition very generic. We need to take ANY kind of message, transform it using ANY map, and deliver it to ANY endpoint (say like a SOAP Web service that we don’t know about at development time, so forget about your cushy “add Web reference”!). Then, in addition, we need to be able to DYNAMICALLY choose things like maps and endpoints. Nothing is known at design time. Is it any wonder that after a few weeks on the project I was wandering around muttering under my breath how “generic” and “dynamic” should not be used in the same sentence? J
What properties go into context? The envelope please…
This is something that will evolve over time (and in fact already has as I am doing some early Iteration 2 work), but what you see below is the starting point:
In the current implementation, properties that are used are:
|
|
|
|
ExceptionAppName |
When creating an exception message, this is for you to store the application name that you will subsequently set a filter for |
|
ExceptionProcess |
When creating an exception message, this is for you to optionally store the process name that you will subsequently set a filter for |
|
MapType |
Fully qualified map name to apply. Any messages that pass through this instance of a CtxSetGeneric component will have this map applied |
|
MapRulesPolicy |
BizTalk rules engine policy name. When executing, the type (namespace+”#”+rootnode) of the source document is passed to the rules engine. You will need to provide rules that return a fully qualified map name based on this information. |
|
StepToDoNext |
If you are submitting a message for transformation, this contains the upper case string “MAP”. The transformation process is decoupled, it will process all messages that have a StepToDoNext equal to “MAP”. |
The list above is what we actually used for Iteration 1. However, while working on that, I was also looking ahead to what would be required in Iteration 2. So, the complete list of metadata properties is as follows (caveat: change is CERTAIN!):
· ContextEnvName
· ContextEnvVersion
· ExceptionHandlerDefault
· ExceptinoAppName
· ExceptionProcess
· EndpointSelAssemblyName
· EndpointSelMethodCall
· EndpointUDDILabel
· EndpoinitURI
· EndpointXpath
· EndpointMoniker
· EndpointMessagingPattern
· MapType
· MapRulesPolicy
· MapSelAssemblyName
· MapSelMethodCall
· MapUDDILabel
· MapXpath
· MapLocality
· StepsToPerform (already changed, details soon)
· StepToDoNext (already changed, details soon)
Getting data into context
Now that we have defined the metadata associated with a message, we need to populate it. For my purposes, I saw three primary ways to fill this need:
- Using a pipeline component
- Using a property schema
- Referencing the schema project
There is a fourth way which we will do in Iteration 2, which has to do with SOAP message processing, and that will be the subject of a future blog posting.
The ESB metadata associated with a message as it travels through the ESB is defined in the EsbEnvGeneric schema. These properties need to be set so that the Core Engine knows how to process a message. The following table shows the three ways we’re addressing here, as well as the point in a message lifecycle where the technique can be applied.
|
|
When a message is first received |
Based on a receive location |
Programmatically in an orchestration |
|
Using the CtxSetGeneric pipeline component to set properties in a receive pipeline
|
Yes |
Yes |
No |
|
Using a property schema inclusion for automatic promotion
|
Yes |
No |
No |
|
Adding an assembly reference
|
No |
No |
Yes |
When to use this technique: When your message originates outside the ESB, and you want to set metadata values based on the message’s entry point into the ESB.
Example: message received from a trading partner, flat file from a mainframe, etc.
Receive pipelines are the “on-ramp” to the ESB from the outside world. If your message comes from outside the ESB, it will enter through a receive pipeline. In this case, you would create a custom receive pipeline that sets the metadata properties in the ESB CtxSetGeneric pipeline component.
With a pipeline component, we have the sub-choice of being able to do “per instance” properties. This means we could have multiple receive locations using the same generic pipeline, but property values would vary based on the configuration for a specific receive location.
BizTalk Server 2004 had the ability to do “per instance” pipeline component properties, but it was not surfaced in the user interface. With BizTalk Server 2006, the ability to set these properties is now provided by the user interface. Simply put, this means you can do runtime configuration of component properties, as opposed to having to create multiple receive pipelines, each with the same component, and hard-coded-at-design-time values.
When to use this technique: When your message originates outside the ESB, and metadata is contained in the message
Example: message received from a trading partner, flat file from a mainframe, etc.
You can add the EsbEnvGeneric schema as a property schema and promote properties in your own schema. This was, when a message is received by BizTalk, those values will be promoted into message context as the values you have specified.
To do this:
· add a reference to the org.xyz.esb.engine.schemas (assuming this matches the assembly that contains the property schema) assembly to your project
· right-click a schema node and choose “show promotions”
· click on the “Property Fields” tab
· click the folder icon to bring up the BizTalk type picker dialog
· select the References\org.xyz.esb.engine.schemas\Schemas\prg.xyz.esb.engine.schemas.EsbEnvGeneric (assuming this matches where you have the property schema) schema
Next, you will need to select the field in your schema that will contain the value to be promoted into a message context property. To do this:
· In the left-hand treeview select the node that contains the value and click “Add”
· In the dropdown list in the Property column, select the property you want the value assigned to
When to use this technique: When you are creating a message in an orchestration, and you need to set metadata before publishing to the MessageBox
Example: you create a member update message, and you want to transform and deliver it
If your application is creating messages in orchestrations that will be processed by the Core Engine, you can access ESB functionality by setting metadata properties and submitting the message to the MessageBox via a direct-bound port.
You get access to context properties in BizTalk expression shapes by using an open-parenthesis after a message, which will bring up an IntelliSense listing of context properties. However, in order for a context property to be shown in this list, you first need to add a reference to the assembly (the reason you see so many by default is that a new BizTalk project will contain a reference to the Microsoft.BizTalk.GlobalPropertySchemas assembly).
Once you have added the reference, then the ESB properties will be available for any message type.
Now that we have the ability to have metadata associated with ESB messages, we can start building out the functionality that makes up the BizTalk-based “Core Engine” of our ESB. Stay tuned!
|