Automation, cloud migration, and cost optimization for a global tech company

Executive summary

Client background

Our client is a global provider of managed cloud services with close to 10,000 employees in offices worldwide. The company offers personalized white-glove cloud hosting solutions to help their customers simplify infrastructure management and reduce the strain on IT resources.

Business challenge

Our client needed to make the process of generating monthly service reports for their clients more cost-effective. The process required lots of manual work and the use of a paid third-party tool. This was caused by the lack of a centralized and unified approach to handling vast amounts of data that the client received from their equipment.

Value delivered

Sparkent has helped the client significantly reduce their expenses by migrating their MS SQL Server infrastructure to GCP and automating the process of generating service reports.

Success story in detail

Business challenge: extra expenses incurred by on-premise infrastructure and a lack of data centralization

Our client required efficient tools for handling vast amounts of data that was generated by their equipment. Virtually all of the company’s siloed data was managed on-premises, which incurred significant costs and made scalability quite problematic. Their sprawling IT resources caused issues with data accessibility, leading to inefficient data governance, storage, organization, etc. This data was essential for generating Monthly Service Reviews (MSRs) – equipment performance reports that were sent to clients every month. As a result, this process was inefficient and cost ineffective: it required lots of manual work and needed a paid third-party tool.

Implementation: data warehouse development, cloud migration, and data quality assurance

Together with the client’s team, our experts have migrated the on-premise infrastructure from MS SQL Server to a unified data warehouse that we have built on GCP. Over 70 operational data sources, 4 data warehouses, and 1 data lake have been migrated and consolidated to Google Cloud. Over 20 servers have been decommissioned in the process, leading to significant savings on the client’s side. Additionally, our experts have ensured high data quality in the warehouse by locating and eliminating anomalies.

Implementation: data standardization and automating service report generation

Our client has over 40 data centers around the world, managing hundreds of thousands of customers’ virtual machines. Every few minutes they receive data from each piece of equipment (servers, routers, switches, etc.), such as CPU usage, memory and packet throughputs, etc. The client used a third-party tool to aggregate all of this information and manually create MSRs for their customers. To make this process more efficient and cost-effective, the client decided to replace the third-party tool with their own solution. The new solution would utilize GCP since the client is an official Google partner.

First, our team investigated the workflow of the third-party tool to learn how the data is retrieved, interpreted, and transformed. After a thorough analysis, we built a new solution that uses Google Cloud Dataflow to transfer data from Kafka to BigQuery. Since the client used different vendors on their servers to collect information (Cisco, IBM, etc.), we also needed to standardize and unify all data. Therefore, instead of using the standard ETL (Extract-Transform-Load) approach, we used the ELT method, where data is first extracted from Kafka, loaded into BigQuery, and only then transformed. This has enabled the solution to create comprehensible reporting tables with information about the performance of each server instance.

Our team has also helped the client streamline the generation of MSRs. The process involved lots of manual tasks and consumed nearly 17,000 work hours per year. Sparknet has helped the client simplify this process by consolidating information to BigQuerry as a single location. Once the data was consistently available across the customer base, our team automated the manipulation of raw data into MSRs.

Value delivered by Sparknet: optimizing operations, automating manual processes, and cutting costs

Together with the client’s team, we have built a unified data warehouse in GCP, allowing the client to elevate data management across the business. Our team has created various new solutions on GCP and maintained the data quality of the migrated data warehouse, allowing the client to benefit in a variety of ways:

  • Optimized costs by migrating from MS SQL Server to GCP and decommissioning over 20 servers;
  • Streamlined data collection, management, and accessibility by centralizing and unifying all data in Google Cloud;
  • Significantly reduced expenses by automating the MSR generation process that consumed nearly 17,000 working hours per year and required the use of an expensive third-party tool.