Overcoming challenges with distributed micro process flows
While building distributed micro process flows, the key set of challenges that are often encountered are primarily around — a.) breaking logical processes into more modular flows b.) traceability & monitoring of the distributed process flows.
In this post, I’ve presented solutions that I have tried and worked out while working on these challenges.
Breaking the monolith
While there are various approaches available for breaking processes into micro process flows, the three key approaches[1] that I found quite effective are:
- Commands and event based execution model where in each of the micro process flows with a given bounded context connects to some kind of bus for all inter micro flow interactions and sends commands to the bus or wait for business events in other micro flows to happen. This enables reactive style event driven process choreography.
- Direct one-to-one communication model in which a micro flow directly invokes the other flow(s) within or outside a bounded context, mostly in a synchronous and blocking fashion.
- A distribution model where in the inter micro flow communication is mediated by another micro process flow. Found this to be very useful while managing activity chains as this is very much in-line with what the pioneer process flow orchestrators (like AWS Step Functions, Netflix Orchestrator and Uber Cadence) have done.
In short, setting the boundary of the bounded context helps to break the large processes into micro process flows there by making it possible to easily react to business changes with quick turnaround times for build and deployment of process changes which in turn finally results in a smoother process flow upgrades for Biz, Dev and Ops teams.
Traceability and Visibility
The other biggest challenge that comes with distributed micro process flows is the complexity associated with coherent auditability & traceability for Biz Ops teams (for instance, how to determine when a particular micro business process flow got executed and in response to what business event? ), lack of end-to-end business process visibility for knowledge workers (How did this micro process flow reach this state and who owns the current business process step?) and ambiguous error handling in case of business exceptions within a bounded context (for instance, what should be done in the event of a micro process flow failure and how should the other micro process flows react to it?).
Micro process flow manager
One to the solutions that works well is to have a centralized micro process flow events manager, through which all events that occur within the micro process flows would be emitted and fed into an events streaming platform like Kafka/AMQ Streams for reliably channeling the process flow events.
This could then be funneled into Elastic stack for performing deep searches and correlation of the events that occur across different micro process flows there by providing the BIZ Ops team with the much needed visibility into the process state and processing metrics, insights and trends at a very detailed level.
Add description
Further, feeding the micro process flow events into Radanalytics spark cluster and leveraging the Machine Learning algorithms would help to
- Automatically determine the next best action for a given process model in real time resulting in an adaptive case management system
- Also, using the intelligence gained from micro process flow events data through Machine learning, some of the human tasks could be fully automated or even eliminated.
- Besides that using the overall process flow events, very insightful heat maps could be created to understand the overall process level bottlenecks.
- And finally, this also paves way for dynamic workforce allocation and optimization for closed loop process improvement requirements.
For instance, using the overall process flow events, very insightful weather forecast type heat maps could be created to determine the time spent at each step with in the overall business process and steps that needs detailed attention and tuning for specific hours of a day. Also, based on a time period specific sudden surge in the inflow of business transactions, a planner engine could be made to automatically kick-in to auto allocate more knowledge workers for that specific step only for a specific period of time from a resource pool like shared services center resulting in dynamic workforce allocation and optimization.
