In my experiences as an EAI consultant, I have come across many people who asked me questions like –
- What is an EAI and Why is it necessary?
- Why not simply follow client-server model?
- Why simple request-response model does not work?
Frankly, it was really difficult for me to convince them with some quick and short responses to those questions. But with some time spent, after explaining them with some proper enterprise use cases people use to get the clear picture and understand the reason. So I thought, I can put my views on this as a post and share it to you all so that it may help fresh integration developers to understand the concept of EAI and how it helps organizations from both Business and Technical aspects.
It is a known and have to be taken for granted fact that applications in large enterprises cuts across heterogeneous systems (like systems to manage Billing, CRM, Stock Handling, Provisioning etc.). This is kind of inevitable and it is very uncommon that we will find an application that runs in isolation. The reason behind this will be obvious when someone understands the simple logic behind it, which is nothing but the necessity of interactions between the systems and data exchange between them. When we say systems, in this context, they could be source systems (Customer Interaction Interface, Point of sale, Kiosk, exposed API etc..), backend systems (CRM, ICMS, Ericsson, SAP, Comptel etc…) and data store/processing systems (Database, Filesystem, Message processing server etc…)
A naïve organization without proper architect may come up with an integration design where source systems interact with rest of the systems directly.
This may look fine when the number of systems involved in the design is a single digit number. But sometimes the number of these heterogeneous systems might go to double digit based on how large the organization is. In such scenarios, following challenges are to be addressed with the prior approach:
- Tight coupling between the interacting systems
- Change management is a nightmare
- Code refactoring is almost impossible
- Huge cost and effort when there is any breaking change in the system service contracts
- Tedious work involved in handling and mapping multiple data formats between the systems
- Adopting new protocols or transport methodologies will cost a fortune
Above mentioned is if there is a direct coupling between 1 source system and 1 backend system.
The effect will be directly multiplied by ‘N’ if the same source system is interacting with ‘N’ backend systems.
This is not over, the same challenge could get more worse i.e., N*N = N2 when more than one (N) source system starts interacting with ‘N’ backend systems
So, what is the answer to this problem? Who can address these challenges and provide a solution?
EAI is a NEED… not an optional WANT….
This is when EAI (Enterprise Application Integration) comes into picture.
Difference between normal application integration and enterprise application integration is that, EAI seperates the concerns about –
- data exchange formats
- target (backend) system details
- methodologies or protocols used for interactions
- platform or technologies used for interactions
from the source system. To do this, EAI used an effectively proven system called ESB (Enterprise Service Bus).
Considering these, EAI could turn our previous monolithic application integration architecture into below:
Here, Source systems ONLY interact with ESB in standard canonical format irrespective of target system data format. It is job of ESB after that to transport the collected request/event from source to target concealing all the details about what transport/technology is being used to do this job.
So, with this in place, typical challenges with normal application integration will be gone:
- Tight coupling between the interacting systems.
Source systems are no more tightly couples with backend systems
- Change management is a nightmare.
Any change in handling data or logic will be now single place, ESB.
- Code refactoring is almost impossible. Code refactoring related to interactions will be at one place, ESB
- Huge cost and effort when there is any breaking change in the system service contracts. All breaking changes have to be now addressed only in ESB which brings the cost/effort involvement from ‘N’ source systems to ‘1’
- Tedious work involved in handling and mapping multiple data formats between the systems. With ESB in place, source systems always use canonicals which are system independent. So mapping efforts are always one to one rather than many to one or many to many.
- Adopting new protocols or transport methodologies will cost a fortune. Any technology or transport protocol changes will now have to be taken into consideration only inside ESB. While, the same have to be changed/ implemented in every single source system in case of normal integration
Now we know why EAI is a necessity.
How an EAI should be…
So, is that it?? Is our integration problem completely solved just by this 2-page documentation with couple of fancy architectural diagrams?
Above EAI pattern solves only half of the bigger problem. Other half of the problem lies among below:
- What about network interruptions?
- How to achieve guaranteed delivery?
- Approach to handle asynchronous scenarios in reliable fashion?
- Administration, standardization and discipline inside ESB
These problems can be solved by adopting the blend of some successful patterns into EAI like Message Oriented Middleware (M-O-M), Event Driven Architecture (EDA), Service Oriented Architecture (SOA) etc.
The proportion of these patterns depends on the type of applications/ interfaces the organization needs.
Message Oriented Middleware fits right into the places where Point to Point interactions does not makes sense. Like in fire and forget scenarios. This makes the application experience faster. All network reliability issues, guaranteed delivery issues can be handled by MOM using retry mechanism and inbuilt short term data stores.
Event Driven Architecture is another successful architectural approach in ESB which is completely opposed to request/response model and everything is driven by an event. While EAI separates the concerns between source and target systems, EDA separates the concerns between everything. All the needed jobs will be events and any interested parties can be consumers to those events. So, no one knows who gives what and who takes what. This enhances the true power of loose coupling.
Service Oriented Architecture helps to bring more discipline, standards and rules inside your ESB. It must be clear and obvious by now that ESB is a common box between all the interacting systems. And it is often common that “common” places like this get chaotic and dirty with time unless until rules or policies are enforced. SOA helps in enforcing such policies in both design time and runtime of application.
I want to conclude this using the same line I started this document with. Larger organizations have to take it for granted that their applications will have to cut across multiple heterogeneous systems. This is inevitable and obvious as every large organization wants their application to be a huge success and for the same reason they have to properly segregate, distribute and manage data in various systems. This distribution involves huge maintenance, proper integration and easy data interactions to make everyone’s effort successful. But this may end up as big failure involving huge cost and effort if proper integration strategy, architecture and teams are not in place. EAI has been proved as a successful application integration approach for many larger enterprises involving enormous number of systems.
EAI also can do its best if right blend of design patterns is adopted into its ESB like MOM, EDA and SOA…