Steering projects; Where is the improved in "new and improved"?

Enterprise architecture is all about improving, continuously. Our target architecture is our aim, and every step towards it may alter the aim. The new aim is a better aim. That is what it is all about: Improving as we walk to a better architecture. The steps in this walk are our projects. We need to identify, define, implement and steer our projects. We totally depend that a project understand is part of the whole. Every task in the project will go in greater detail. These details will bring us a better understanding; either some part of the goal of the project is a bad path, or it is a good one. Maybe the traversal of the projects tasks really find a golden egg, or a pit of snakes.

Projects live a life of their own. The team is dedicated to their goal, and they will continue to walk within the mandate and quite a bit of self-focus. Very uncommon, or never, has a project said to the steering committee "our task is unnecessary, please stop this project". It is not that the project not is trying to reach its goal, or being loyal, the team is honest in doing what they believe is correct. The purpose of the steering committee is, well, to steer. The steering committee in this context is any entity who is responsible for steering the project. I remember a project manager once said that informing the steering committee was like growing mushrooms "keep them in the dark and feed them horse shit". And maybe this illustrates the challenge; the projects must not obey the project-mandate or architectural blueprints alone.

In this context the mandate must be extended with the target architecture and a description of what the project must participate with. This includes components which are there already - they may be extended - , and new components that later projects will benefit from. The later is a tougher one. It is then vital that projects understand the purpose (why and for what) of this component in the target architecture, and have a steady set of principles. The message must be repeated and repeated, the enterprise architects must be absolutely sure that the architects in the project understand why. Remember that experience the project gain, may eventually alter the target architecture, there must be close cooperation between architects at many different levels.

So this brings me back to the headline: "where is the improved in "new and improved"? When we staff projects, most often we choose people who know the functional domain and someone who know the new technology. And what do you get? To often the existing domain in a new technology. Does this bring us closer to the the target architecture? Can the steering committee see what is coming? Too often they haven´t.


In addition I think the overall process must be agile so that the project mandate and budgeting may be adjusted along the way. Also we have the good opportunity of having project demos as the sprints go by. This makes it easier for the steering committee and the enterprise architects. I think Gartner has coined this overall process EAD "Enterprise-class Agile Development". Using project as changers on the path to an improved enterprise architecture.

All you get is new, but not improved, if you don't make sure that the project does understand the purpose of the target architecture.

Steering projects; Where is the improved in "new and improved"? by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Requirements are long lasting

I have working with large systems and maintenance for a long time, typically within the Enterprise Application domain. I have worked in projects where the code should be "self documenting" and projects that follow RUP diligently. The first one became unstable after a while because of unclear requirements. There was too much code inspection to explain the functional owners what the system did, and there was a problem finding out if features where a bug or not. Also re-factoring was hell. The second one we spent too much time not writing code, and the system really did not reflect the formal requirements. It was just too much documentation to put up with. By experience I think that there must be an information model (and some Glossary or Concept descriptions), description of behavior (eg. Use Cases), and sequence or activity description tying this together. I have also worked with TDD (or similar where tests are written in pair with the code) and seen what a great effect this has on good code modularization.

I have no experience with user stories, but i see and hear (together with BDD and TDD) that it is a good approach to requirements handling that fits the backlog and suitable for the sprints. And together with a suite of functional test, it serves a solid combination for the project delivering an application. My concern is that developing the system is only approx. 10% of the lifetime cost (or less). During the lifetime there will be different system owners and developers and they need to understand how the system should work. There will be reviews and overall planning later on that need to understand the system without looking into the code. In such a context the user stories seem to fragmented and seems more like a changelog for the development, rather than a description of the system.

For the last 90% of the lifetime there is a need for a robust requirements base. How do we ensure that this is maintained? I do believe (from experience), that the requirements must be described when it is developed and be aligned with the version is released. At that time I also think that the functional parts should be a whole more as defined by good Use Case practice. Regardlessly of how you wish to document the requirements, at least kept them in a common and persistent store (not post-it notes), so that there is a stable base for future reference.

And also; it is hard to keep such documentation in place, it certainly requires diligence, obligation and thoroughness to keep up.
In a development team I believe that someone must be responsible for the overall functional package, and that is is not just up to each developer to write user stories. All to often i have seen that functionality is written again without reusing or extending what has already been developed.

Requirements are long lasting by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

New tools, old methods. A masons story

We got all these new tools, but we must also change the way our developers and architects solve problems with it. We got to understand new tools and methods to actually make better software systems. Eg. Classes with 6000 lines, bad separation of concern between layers or components, no clue of MVC, no interfaces, etc.

As an example from constructing a house:
You have descided to construct a house in wood. All your requirements point in that direction. But all you got are masons. How do you think that house would look like without the proper training (or on-site inspection of an existing wooden house)?

A house in wood is constructed very differently than a house in bricks. You first build a frame with beams, then insulate and cover with boards.  The mason would probably lay every bean horizontaly with mortar in between. It would surely be a costly house, and would have none of the qualities we originaly specified this to have. Also the masons would think you where an idiot.

New tools, old methods. A masons story by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Java is mature. The software-teens are over.

Just a few words on why we would expect less expensive maintenance and better quality in our software systems by using Java and following the standards (de-facto or formal). And also on why I think Java is mature and will not go away for some time.

I would like to compare with the construction of things. Until well into the industrial revolution, every part of something (eg. a clock) was made by hand. And every craftsman had his own approach to this, and it was labor intensive. Screws for example had all sorts of screw threads with different angles, lead, pitch, diameter and handedness. This also meant that if something broke, you had to get that exact copy, probably from the same craftsman (or his son...), no other screw would do.

The same goes with construction of houses. Well into the 50´s there was no obvious standard. Every window in a house was tailor made, they looked alike but had small deviations due to optimizing the usage of materials. The size of a beam was constructed from the "feeling" of the carpenter and the available resources. Standardization has brought many good things into play; training of carpenters, estimation of cost, material and labor, lower prizes due to mass-production; everything is built in 60cm sections.

Now what has this to do with software? Most of today's legacy systems (all the way from the early cuts on software development) where built in a time or with a technology where everything had to be made from scratch. There where virtually no components and there where no good technology for using components. Also training was bad, Computer Science was in its infancy. Every programmer mastered its own discipline, and strange arguments could be heard in the corridors; should arrays start with 0 or 1; big-endian or not, new line before or after comma? These "great" discussions are really a waste of time, the real value is not there. It is in standardization and doing things so that others can do maintenance and understand what the system does. But as with the screws and windows; doing is learning, is was the software-teens we had to get through.

These legacy systems work, but only after a close study, and they will always be hard to maintain, just because everything is tailor-made. If it was a building it would either be condemned, or it would be protected by the antiquarian.

Recently we had a PoC where we found that validating an xml-file (by streaming through it) was just some lines of code. Various legacy versions is much larger. Which do you think gives the most cost-efficient maintenance and ease of change? Which is most robust as to change in who does maintenance on the system? Which one is faster?

...
SchemaFactory schemaFactory =
SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(new File(schemaFile));
Validator validator = schema.newValidator();
validator.validate(new StreamSource(new BufferedInputStream(new
FileInputStream(xmlFile))));
...

I think after approx 60 years of software development and now 15 years of Java (that was 45 years without), we see a very mature language, a great set of standardized technologies, a vast number of components and systems, a large community and supplier base, and a resilient run-time architecture.

I can't see that we could make a better choice.

(We are seeing new efforts into more efficient languages, but for now they are niche players. If a new language should replace Java, it would probably take 10-15 years of maturing before it could actually be there. Just think of all the work put into the Virtual Machine... And as I have argued; the real value is in how we actually construct software systems, not the language by itself)

Java is mature. The software-teens are over by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Workshop with John Davies

We had John here for 2 days, it was intense and very encouraging. First and foremost we got support for our efforts in the direction as to what stack of technology to use, what development process to follow and how to construct future applications. Secondary we got tips of directions or prioritization to take, that just seems so obvious after the session was over, but that we did not see before we started. Third there are feedback and experiences that we not yet know are relevant or achievable for us.

We started off with some hours of introduction to our domain. It is a steep hill and I am sure we could have used weeks to get more into it. But still I think we managed to get the rough picture so that we got input targeted for our challenges. Compared to what John previously has been working on, I think roughly that we have a lot less transactions, but the diversity in data captured and in regulations put on our systems seems harder.

So to shortly summarize:

• Get confidence and commitment from top management (by demoing prototypes).
• Build prototypes, have a sandbox, prove concepts. Don´t use valuable staff on reports.
• Run agile, have much shorter projects.
• Any plan over 2 years is a risk or a waste.
• Understand your problem, use what you need, not what a salesperson think you need.
• Implement software to run on more that one platform. Put this into the build process, be able to change database, app server or cache at any time.
• Don´t use to much time on storage architecture. Build for the cache, concentrate on the business logic.
• Integration architecture is much simpler than what´s in an ESB product; Make protocol-neutral services, use a broker and DNS (JNDI).
• Standardize and virtualize IT platform.
• Use UML in design.
• Ruby on Rails is great for GUI.
• We have the chance to make a revolutionary tax-system.

Are you up to it? Can you help us implement this?

Comment on: RESTful SOA or Domain-Driven Design - A Compromise?

I find this talk by Vaughn Vernon very encouraging and it fits nicely into what we are trying to achieve.
It is very clear and educational and should be made basic training at our site.
http://www.infoq.com/presentations/RESTful-SOA-DDD


It actually fils a gap very nicely as to how our target architecture may be implemented, and more importanlty; why it should behave as as described. The Continual Aggregate Hub and restaurant is really a macro description of how such domain systems may cooperate in a pipeline, and how they can share a document (this is how we store Aggregates) storage.

In in the migrate to the cloud I am talking about ´Cooperating federated systems´; and this is exactly what I observe Vernon is talking about in his presentation. I especially like the Domain Event (page 19 and 34), it is the implementation of what I have called the ´Lifecyle´. Maybe lifecycle is not a good word, but it chosen to illustrate what drives behavior on the domain objects, between and within, these loosely coupled systems (they have separate bounded contexts).

The Domain Event Publisher (page 37) is the Process-state and handling of the CDH, and it is relaxed so that subscribing Modules may consume at a pace they wish. The Event logger is also an important aspect of it so that re-running event is possible. (illustrated nicely at page 39-).

We have very interesting performance results (over 50.000 aggregates pr second) from a Proof of Concept on Continual Aggregate Hub running it in a grid.

(BTW: The real essence is not RESTful or not. Other service implementations and protocols will also do)

Aggregate persistence for the Enterprise Application

Designing Aggregates (Domain Driven Design) is key to a successful application. This blog explains what the Aggregate and the Root Object is like in our Continual Aggregate Hub.

As I have written in earlier blogs, we are trying to handle our challenge - a large volume, highly flexible, pipeline sort of processing, where we seek to handle quite different sets of information, fix it and calculate some fee - by mixing Domain Driven Design, SOA, Tuple Space, BASE (and others) and a coarse grained document store that contains Aggregates (see previous discussions where we seek to store Aggregates as xml documents).

Known good areas of usage
We know that a document approach is applicable for certain types of challenges; Content Management Systems, Search Engines, Cookie Crunchers, Trading etc.. We also know that documents handle transactions (messages) very nice. But how applicable is it to an Enterprise Application type of system. We want loose coupling between sets of data because it can scale out, functional loose coupling and for other reasons discussed in earlier blogs here.

Why two data structures?
We want systems that are easier to develop and maintain. Today most of Java systems have one structure on the business layer, where we successfully develop code and have a god pace. Using unit tests and mock data to enable fast development. Every thing seems fine, until we have the object relational mapping (ORM). Here we also must model all the same data again, but now in a different structure. At the storage level we put tables and constrains and indexes so that we are sure that the data is consistent. But that also has already been done on the business layer. Why do we continue to do this twice?
The relational model is highly flexible, and is sound and robust. A good reason to use it, is that we want to store data in a bigger context than the business logic did handle. But would´n it be great to relax this layer and trust the business logic instead?

Relational vs. Document
We know that the document approach scale linear very well, and that the relational database does not have the same properties because of ACID and other stuff, but why is it so?

Structure Comparison

The main reason for not being able to scale out, is that data is spread out over many tables (and that is the main structure of most object databases too). Data for all contexts is spread on all tables. Data belong to Party S is all over the place, mixed with Party T. During an insert (or update) concurrency challenges happen at tables A, B and C. The concurrency mechanism must handle continuous resource usage on all tables. No wonder referential integrity is important.
In the document model the objects A, B and C are stored within the document. This means that all data for Party S is in one document and T is in another. No common resource and no concurrency problem.
The document model is not as optimal if there are many usage scenarios that handle all objects C, regardless of what entity it belongs to.

The Enterprise Application challenge
So how do we solve the typical Enterprise Application challenge, with a document store approach? (Should´n we be twice as agile and productive, if we do not need to maintain a separate storage model.) Finding the granularity is important, and most probably should follow the main usage scenarios. To be able to compose aggregates there should be some strong keys that the business logic must ensure referential integrity on. Even though we may not have integrity checks in the storage layer, I am not sure it is that bad. We do validate the documents (xsd and business logic) before we store. And I have no counts on how much bad data I have debugged in databases, even though they have had a lot of schema-enforcement.

The super-document
A lot of the information that we handle in our systems are not part of the domain. There are also intermediate information, historical states and audit for instance. Remember that we in a document approach reverse the concept and store everything about Party S, by itself in its own document. To be able to cope with a document approach, the document itself must be placed within a structure (a super-document) that has more meta data about common concerns such as: keys, process information (just something simple like : new, under construction, accepted), rationale (what decisions did the system do in order to produce this result), anomalies (what errors are there in the aggregate), and audit (who did what, when). The <head> is the Root Object, and its generic so that all documents are referenced in a uniform way. The super-document is structured like this:

<head>
<keys>
<process>
<aggregate>
<rationale>
<anomalies>
<audit>

The IO challenge
We succeed in this only if we also manage to make this perform. The main pitfall for any software project is the time and space dimensions. Your model may look great and your code super, but it does not perform, it does not scale, and you loose. The document storage model is only successful if you manage to reduce IO, both calls and size. If you end up transporting too much information, or if you have too many calls (compared to ORM), then the document model may not be optimal. An Enterprise Application may have 100´s of tables, where probably 30% is m-m relations. I have seen applications with more that 4000 tables... Only a genius or semi-god may manage that. Most probably it will just be unstable for the rest of its life-time (see comment on the Silo). My structure example above is way too simple compared to the real-world. But surely for many of these applications there is a granularity, that fits the usage scenarios better. I have seen documents with 100.000 nodes getting serialized in less than a second. Does not 20 document-types, small and large, seem like a better manageable situation, than 200 tables?

In our upcoming Prof-of-concepts we will be investigating these ideas.

Aggregate persistence for the Enterprise Application by Tormod Varhaugvik is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.