Data Integration Layer:
The foundation of our architecture lies in the Data Integration layer, responsible for seamlessly merging data from various sources. For real-time data streaming, Kafka is employed as a robust software solution, ensuring continuous and reliable data flow. To integrate with diverse data sources using different protocols, connectors, and technologies, Kubernetes (k8s) and Camel are utilised. The integration supports SQL databases, Secure File Transfer Protocol (SFTP), Hypertext Transfer Protocol (HTTP/HTTPS), and Java Message Service (JMS), providing a comprehensive solution for data consolidation from varied platforms.