The purpose of this project was to bring the data scattered across 180 databases to one single platform for providing a holistic view. Our client, a leading IT company, provides services to UK government for its operation in more than 180 countries using different applications.For each country a separate database was being managed and maintained at different data centers across the globe. Reporting mechanism was in place for each country to provide Management with parameters which can help them to make effective decisions. The Reporting mechanisms, however, failed to achieve the purpose of providing complete and comparative view of each country’s operations with respect to other countries; as the number of reports was specific to a country and the reporting was spread across 23 such reports for each country. Having clear insights as to where we stand in terms of organization objective and taking corrective measures to move towards achieving it was only possible when data of all these different countries could be consolidated and analysed on different parameters. This was currently not possible in the existing situations. Client realized the need and engaged our team for the project.
We analyzed all 180 databases and studied underlying schema structure in record time of 15days. We designed a single schema structure which can accommodate the need of all 180 databases. We implemented scheduled tasks running at regular intervals to import data from different countries databases to 3 central repository databases using Python as the scripting language. We created web interface to facilitate the client for generating parameter based report. Client was happy as the implemented solution provided him consolidated views across countries, improved efficiency and reduction in expenditure cost.
A leading IT company which is part of a 2-billion-dollar group and specialize in handling Government Projects
Customer was running 180 databases, 25 application connecting these databases, 30 persons engaged for managing reports for 40 countries, with 30% delay and 5% errors. No solution available with the Management team to run report based on parameters. Instead Management team had to look out for reports from the emails.
We analyzed all 180 databases and studied underlying schema structure in record time of 15days. 3 central database repositories were created based on application version which was runningat various data centers. Customized ETL application was developed to collect data from various database.Jasper report tool has been used for report development.AWS cloud has been used toensure horizontal scalability and ease of sourcing infrastructure. File based data transfer was developed to ensure fast processing of data at file system level. Data is growing with rate 3 GB per day and processing time was critical as window for processing data was only 4 hours.