Showing posts with label XML. Show all posts
Showing posts with label XML. Show all posts

Saturday, March 7, 2015

Proven architectural properties

We are now in production with our new scheme as of 2015; Ongoing reporting of wages live from all payroll systems around the country. My blogg has been about design ideas of 2010 and the demanding process of transforming our systems, knowledge and organization from silos to robust scalable 24/7.
We have managed a fundamental modernization based on a paradigm shift in software design and implementation

We have had a smooth start with great results:
  • Of 223.842 deliveries 97% get response of <15 seconds (end-2-end)
  • 24/7, peaking at 3.190 deliveries / hour with constant response (a delivery is all payroll from all employers)
  • "Micro services" architecture inspired by "Reactive Manifesto»
  • Changes and fixes rolled out continuously in the daytime, in practice 2-3 times per week
  • All business logic running live, also with consistency against previously reported data
  • Mainly Open Source
  • Our motto: "Simple, testable, scalable" (as described in previous bloggs)
  • Java version: 1.7 (soon Java 8)
  • XML: Built in JDK
  • REST: Jersey, Apache Http Client, Atom Feeds
  • Clients: HTML5, CSS3, Backbone.js, Angular.js
  • Servlet container: Jetty, Grizzly
  • Parallel handling: Akka
  • Engine: ElasticSearch
  • Distributed Coordination: Zookeeper

Proven architectural properties
The design from 2010 has proven abilities. Wise choices in 2010, broad organizational anchoring and continuous IT architecture and technology management, as well as trial and error has made this possible.
  • Few consecutive errors   
    Proves the ability to change. The system is understandable and we have control of it.
  • Frequent deliveries
    Ability to change. Quickly correct errors and introduce new functionality.
  • The performance was very much improved in a short time
    Scalability. "Design for the cloud"
  • The systems moved (unchanged) of the convergent platform within a few hours
    Lifespan principle. Open standards provide portability and market access
  • Solution in operation after 17 months
    Reuse. Cost effective and risk mitigation
The enterprise architecture process
"To put an idea out in life is a struggle for life. You must have steadfast faith in what you are doing"(Klouman)
  • Dare to be politically incorrect
  • Convince even the lagers
  • Even harder; make sure they understand
  • Anchorage target architecture at the topmost level, but every week you have to fight again with rounds of discussions (others or your lack of understanding, or political opposition)
  • Credibility; You must combine "listen and learn" with "motivation", otherwise you will be ignored or "eaten alive"
  • Dream Situation: Make sure that the purpose of a design is understood, so anyone believing in it will make it their own and improve beyond you own capability

Fun and scary statement on the way:
  • "We solve this with SOA, WebServices, ESB and BPEL»
  • "Employers are customers, purchase CRM system»
  • "XML has been tried out in Denmark, and it does not work"

Tax Norway is a large governmental organization of 6.500 employees, this type of transformation is demanding. Thank you everybody.

Creative Commons License
Proven architectural properties by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Saturday, October 4, 2014

The Case Container

This article fills in the last piece, and completes previous articles (linked in this text below) of how we structure Case Handling. It is the design of "Process state and handling" of Continual Aggregate Hub article of 2010.

At the core
A central part to any typical Enterprise Application is the Case or Dossier, and the process handling this. The information going in to a Case, the business logic applied to it, and the subsequent business decision(s). It all has to be filed with accuracy. Case handling get complex because information changes over time, business decisions are made, and the business logic and the information going into it are also complex. Just look at financial institutions and insurance systems, as well as government systems. These have a load of legislation and business rules - that change over time – and every business decision must comply to the rules and information that was valid at that point in time. Otherwise that decision does not have integrity.
Legislation states that filing in such systems must be applied after some sort of standard; in Norway it is called “NOARK”, in the UK “RMS” and in the US “NARA” (and related).

Immutable xml-documents
In our systems design we already have stated that business decisions are immutable, and are stored as XML documents with a set of metadata. The design of the CAH has proven to have excellent qualities with regards to integrity, ease of maintenance and ability to scale. This component is now in production (see BIG DATA - Definite content).

Searching for the answer
We used 3 years looking for this design. And we started out in the BPM (Business Process Management) area. Now that we have the design, we realize that BPM is the wrong end of this challenge. There are a lot of BPM products out there, but they miss out on one main point. It is not the processing framework that is the main challenge, but the Case content (information and rules) and the state of the Case itself. That is why most of these effort are hard to maintain, and often could be solved in a much simpler way. A key point for us is that any Case may be reopened for the next 14 years. We do not want to depend on any specific BPM product, as it is a vendor lock in (enterprise wide process handling).

The missing piece
Let me introduce the CaseContainer, a generic container representing any Case in our domain. It is also an immutable xml-document, and acts as a “binder” between the Case Type and its metadata, the information going into the case, the business logic applied to it, the process state the Case is in, the resulting business decision, and as access control.
It is basically divided into two parts: 1) a set of metadata describing the Case, and complies the NOARK-5 standard. And 2) a set of references to xml-documents it consists of. The document references are divided into two groups: documents going into the case and documents representing the resulting business decision. The metadata makes the CaseContainer be part of a hierarchy of cases, is described by the Case Type, and it describes the case state; both general (eg. Open, Pending, Closed, Reopened) and specific to the case type (eg. The states of this specific case handling process). The Document Archive would be the CAH with its typed documents, or a Archive of binary documents such as PDF (for manual read by a case handler). It is all implemented with REST and XML notation. You can’t find anything more open and robust.


The Case Container described (main attributes, there are more):
AttributeDescription
UUIDUniversal Unique ID for this document
PartyIdThe Id of the party this case concerns
Timestampnil
Case numberThe number of the case. Most often this has some formatting case year, sequence number and version (eg. 2014/45235/4)
Case titleThe title of the Case (eg. "Tax case 2014")
Status codeGeneric case state that complies to your local archiving standard (new, open, pending, closed, rejected...)
FolderThis is also called “File”. It is structured as a flat “namespace” notation to contain “Fonds”, “Series”, “Class”, and “File”. And any hierarchy between them (eg: no.nta.folder.tax.2014)
The Case Type may be the whole or a subset of the "Folder".
StateIt is structured as a flat “namespace” notation to contain any state in any process. (eg. no.nta.process.tax.2014.complaint)
Case responsibleThe case handler or organization responsible for the decision. (eg. "no.nta.authority.username")
Date of invocationThe date for opening this Case version
Date of decisionThe date for closing this Case version
Incoming documentsA list of references to all documents going into this case. We have a unique reference to every version of every xml-document. Usually this is called “Registry entry type” or “Document object reference”.
Business decision documentsSame as above, but here is the list of the documents produced in handling this case. Often these are outgoing documents and can contain an Invoice or a written statement.
AuditA list of all changes made to this case since version 1.

[*) NTA is Norwegian Tax Authority]

Processing the Case
The processing framework handling this is also simple. A Case Type has a set of documents that the Case can consist of. When a new document comes into the CAH is it pre-defined what type of Case should be opened. The Case Type and its State is linked to a processing URI that handle it (see Enterprise Wide process handling for details in how we cope with event driven processing). In that way we connect the valid business rules to this Case Type. Connected to the event is a reference to the Case Container, and its is the Case Container that is transported to its Module for handling. The Application Layer (defined in DDD) then starts processing on it. Any open Case will either be handled automatically or show up in the Task List for manual handling. It will be handled manually if the business logic states so (by some validation), or if some of the documents going into the case is not strongly typed, and must be read by a human.
Processing may also be restarted just by creating a new version of the Case, for example when your code has been fixed and you need to ad-hoc re-run the business logic. This will result in a new version of the Case Container, and a new version of the business decision document. Handling Cases where there is new information, but no new decision, is archived as a new version of the Case Container, referencing the new incoming information, but maintains the last business decision. (see the Restaurant and the role of the "waiter", he is the one composing Case Container when events happen, at a timed event or at ad-hoc need for running a process)

Access control
Security and access handling has a better chance with this design. The user is linked to a set of Case Types and hereby valid Document Types. Now the XACML access control needs only check if the user has access to the Case Type or the Document Types. The metadata containing Case Types and Documents Types is now a PIP (Policy Information Point) for the XACML's PDP (Policy Decision Point). The access control has a high level of granularity and the makes it possible to understand and configure. An otherwise tough challenge in a service oriented environment.

Separating concerns
The CaseContainer is a design that separates the business information and logic, from the decisions made over time. In most systems these are intermingled. Now they form two separate implementations, making the Case Handling System much easier to implement, understand and maintain. The Case Container runs in the Application layer, while the Business logic works on the information in the Domain layer (as defined in Domain Driven Design)

I hope this helps out in your quest. Separating concerns is really what you need in a complex domain.
Creative Commons License
The Case Container by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.