Enterprise wide business process handling

Introduction
(October 2014. This article is updated with the Case Container)
Understanding the requirements, functional and non-functional, is vital when the architecture is defined. The theme in this article is about enterprise wide business process handling. The challenge is to make different types of systems participate in the business work flow and understand what happened for later insight. BPM, Business Process Management, is a huge field with many tools, notation forms, patterns etc. (See process handling in Continual Aggregate Hub and cooperating federated systems). (A system in this text is the same as a Module, although it may reflect that our current systems are not good Modules. We want our current systems to cooperate as well as our new Modules.)
We have chosen to think Domain Driven Design and to elaborate on these main issues:

• how to automate the business process between cooperating Modules
• how to ensure that some manual (human) task is done
• how to achieve high degree of flexibility in changing business processes
• how to know the current state of a Party
• how to know what has happened to a Party
• how to measure efficiency in the business activities over time 
• how to make very different types of systems cooperate

 And some non-functional requirements:

• how to handle the operational aspects over time (batch vs events)
• how to best support that a manual task is now automated
• how to make it testable
• how to have it scalable
• how to change and add Services and Modules without downtime
• how to change process without downtime
• how to re-run a process, if somethings have failed for some time
• how to handle orchestration (defined process) and choreography (event drive) side by side

Observations of SOA tool state and of integration patterns
I will not discuss this too much, but just summarize some observations (I'll get back to how in some later blog):

• In-flight processes is a pain
• Long lasting processes is a pain (both: software needs update now and then)
• Often not loose coupling between process and the state of some domain entity. A lot of processes occur and function within the domain ("make-minor" approach by some process-system is not good)
• Processes are hard to test
• Scaling and transaction-handling gets complex
• Tools have too much lock-in
• The promise of BPEL visual modeling to communicate with business fails
• The canonical integration pattern often leads to context bleeding and tight coupling
• The aggregate integration pattern often is a sign of complex integration, that probably should be addressed with a system by itself
• Business process state is hidden, and history of events is lost or is drowned in technical details
• Message brokers are great to move messages, but bad in history, and not a good tool in operational flexibility
• Too many parties are involved so that maintenance gets slow

Our logical 5 level design
Main goal is to have full control of all events that flow between cooperating Modules, but not achieve an uncontrollable event driven system. (An event-driven system may just as well be diagnoses as "attention-deficit/hyperactivity disorder (ADHD)" system.)

• Enterprise Process Log
  This is purely a log component, with services for registering business and answering queries about them. It has no knowledge of the business process, but of course has some defined keys and types that defines a valid business event (a business activity leads to a business event). It is the Modules that emit domain events to the log component, and the domain defines what the events are. It is a place to store the business level CQRS type of events (or Soft State if you like from BASE), the more detailed events are kept the Modules. Any type of system can emit events either live or in large batches. The implementation effort to send events is little, and the events may be informational and not necessarily used in an automated process. This log will give insight into what happened with a Party. The lifetime principle is taken care of, as this log must be very long lasting. So we want it as simple as possible. (see The BAM Challenge)
• Global Task-List
  This is a simple stand alone task-list whose sole responsibility is to assign tasks to groups of Case Handlers (CH) that do manual tasks in the enterprise wide domain. The task-list has no knowledge of the overall process. The tasks the task-list receive contain enough information to explain what the task is, and to redirect the CH to the right GUI for doing his task. The tasks are defined and maintained in the Module that has a need for manual work, but dispatched to this common task list. These tasks are tasks that are well defined in a work-flow. When a task is done, or no longer relevant (the Modules that owns the task decide), then the task is removed from the task list.
• Process Flow Automation
  When we first have the events in the event log, automating the next step is quite easy and can be done in a lightweight manner. A simple Message Driven Bean may forward the message in a event driven manner via some JMS-queue, or a large legacy system may query for a file once a week because that is the best operational situation for it (operational flexibility is also discussed in CDH). Also events may be logged that only come in operational use a year later, making maintenance flexible, and history robust.
• Case Container
  This is discussed in this article about a generic Case Container representing all Cases handled in our domain. Its purpose to to contain the Case with all its metadata, the process state and references to all incoming and outgoing xml-documents.
• Case Handling System (super domain)
  This is usually called Case Handling and consists of a case which is outside the existing Modules or systems, and that need the completion of many formal sub-processes, but at startup of the process it is not possible to foresee how the case will be solved. This is typically where know business process ends, and a more ad-hoc process is applicable. Also this system support the collection of different types of information relevant for such case-to-case systems. This information may very well be external information collected in a manual matter.
  (by 2014 we have not gone any further into this part of the overall design. It seems that the CAH and the Case Container is sufficient)

Above is the logical design, and this is what we think we need. You might say it follows a hub-spoke design, where the Modules are the spokes and these 4 elements comprise the hub. These are all 4 discrete components that interact in a services oriented manner, with each other and with other Modules or systems. The main idea is that this will enhance maintenance and reduce the need for customizing COTS.

Ill 1. Basic flow and service orientation

Illustration 1. Just to show a fire-and-forget situation (green) where the Tax Calculation and Collection are interested in House events. Tax Calculation wants them live, and the processing forwards the event, while Collection every 2 months, via file, and then issues service requests (yellow) to Party to get details. The EPL event holds a reference to the Case Container. The notified Module then uses the Case Container to open all relevant information for this Case and the process state the Case is in.

Ill 2. Application layer interacts with EWPH

Illustration 2. Show a Module of DDD where  the application layer interacts with the enterprise wide process handling. The green line show how the Case Container is shipped to the Module where its reference are opened by the Application layer and sent to the Domain layer for handling. The Case container is really the container that it is in the logistics world; bringing a complete set of good from one place to another.


We are implementing this with REST and XML, where feeds play an important part in transporting events and data. URI represent addresses for Modules, and are linked to specific Case Container Types.

We do not mandate the usage of the Task-list. If there is some need for task-list internal to a Module that is more efficient for the users that handles tasks solely in a Module, it is OK when it gives a better maintenance in the long run.
Also we do not mandate what technology that does different work in the Process flow automation, it may be Message Driven Beans forwarding to some queue, Mule, Camel, BPEL for some orchestrated process, or simple Query to file.

Design and overview
Of course you still have to have a good design and understanding of the business processes. And it must be there to communicate between business and IT (BPMN is great for this). But as in other areas of systems development: Design is to communicate between people, and implementations is to communicate to machines. Therefore a combined design-implementation (eg. BPEL) will have a hard time achieving both.
The business process is not fragmented, but I argue that the implementation of the business process is best handled in the above-mentioned manner; The process will sometimes occur within a Module (system), and sometimes in-between.



Enterprise wide business process handling by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.