Nielsen - VADE - Case Study

At the end of 2018, the Nielsen company contacted us with regard to the issue of long processing times for batch data analysis and verification tasks. The problem concerned one of the business processes in which the result data is regularly transferred to the final customer. Final verification, both technically and domestically, must take place before handing them over. The Data Science team at Nielsen is developing many advanced mechanisms to validate such collections and are carried out using dedicated applications. They process very large data files, and the analysis itself requires a great deal of memory and processing power. An additional challenge was the need to perform many validations in a short space of time at the end of each reporting period. In conclusion, the problem was that the existing solution could not handle the conversion of all data sets in a given time - it simply took too long. During a workshop with the Nielsen team, we created the concept for a solution that allows existing applications to be nested in so-called “workers”, which are then queued and run on multiple controlled servers. Finally, the results are returned to the management process. The number of execution environments and simultaneously performed “workers” can be adjusted to specific needs. Our architects and the client proposed a specific implementation method, from here we moved onto the programming phase. An experienced team of programmers built an engine under the name VADE. It can be described as an environment for queuing and processing batch tasks in a private cloud. For several weeks Nielsen was testing the solution in one territory. Currently, the engine is being implemented in several dozen markets as a global platform. An interesting aspect of this project was the cooperation between the Data Scientists team from Nielsen and our architects and software engineers. This resulted in a large number of ideas and an efficient solution. The project was implemented iteratively. In the initial versions, we worked mainly on the requirements specification and prototype, then on subsequent functions. Thanks to this approach, the client was able to verify the compliance of the delivered components to their needs on an ongoing basis and use the applications for subsequent attempts. All of this made us reach a working solution within a dozen or so weeks of the reported demand and problem. In our opinion, it is a good example of cooperation between a large organization and an external software house in improving processes.