This reflection project is about real time data migration from an on premise databse to AWS cloud using AWS DMS and SCT. I use AWS IoT device simulator to replicate on premise data center infrastructure by ingesting real time IoT based data. I used Infrastructure as code (IaC) practice using AWS CDK to create, provision and manage resources.

My simulated customer here is The O2 Arena in London. It is a popular place for concerts. The O2 Arean has many shops inside. We call them tenants here. O2 Arena wants to monetize the crowd density and demographic information depending on event for improved marketing and also crowd management.

O2 Arena uses fixed sensors to measure real-time crowd flow, detect sudden bottleneck surges at access gates, and monitor zone occupancy levels safely and anonymously using Smart Venue Infrastructure Hardware. The hardware is mounted on the walls and walkways around the complex.

The Edge and Intake Infrastructure Architecture:

To mimic the physical hardware footprint of Smart Venue Infrastructure Hardware, I engineered programmatic edge simulator using Python's threading module. IoT policy - Devices are strictly limited to iot:connect and iot publish actions on an explicit MQTT topic channel. Services used here are - AWS IoT core, Kinesis Firehose, AWS Lambda, MariaDB, AWS S3, AWS Secrets Manager.

This pipeline kicks off with a multi-threaded Python simulation script acting as an AWS IoT Device Simulator, broadcasting high-frequency, real-time crowd motion telemetry over an encrypted MQTT connection. AWS IoT Core catches the incoming sensor data streams and automatically routes them to Amazon Kinesis Firehose. There the incoming stream data is batched, compressed, and dumped as immutable raw logs into an Amazon S3 Backup bucket for Disaster Recovery Planning (DRP). Kinesis also triggers a serverless AWS Lambda function (MySQL writer) that unpacks and process the streaming payload records. The Lambda function fetches the required database credentials programmatically from AWS Secrets Manager and writes the processed data directly into a MySQL/MariaDB database isolated inside a secure Private Subnet. For administrators needing to query the isolated database locally, an encrypted tunnel is established using AWS Systems Manager Session Manager through a secured EC2 Bastion Host. This allows safe connection to external SQL Client without opening any public internet ports.

The Migration Workflow:

Below I show how second half of the architecture connects to the first half. I perform migration in the second half

Services used here are AWS Glue, AWS SCT, AWS DMS, AWS Timestream, AWS S3, AWS Secrets Manager, AWS RDS, AWS Athena, Aurora Postgres, QuickSight, AWS SSM, Apache Spark. AWS DMS (Database Migration Service) can handle both homogeneous migration (MySQL to MySQL RDS) and heterogeneous migration (MySQL to Aurora Postgres) using the AWS SCT (Schema Conversion Tool). It securely reads the row-based transaction logs from the on-premise simulation and streams real-time database updates over to a managed cloud target (AWS RDS MySQL) while concurrently logging raw data straight into an Amazon S3 Raw bucket. A serverless AWS Glue Apache Spark compute engine reads those raw data dumps out of the S3 landing zone and runs distributed, in-memory ETL transformation jobs to clean, structure, and partition the files. The Glue pipeline bifurcates the transformed outputs by writing optimized columnar files into an Amazon S3 Enriched bucket for historical analytics and pushing active metric rows into AWS Timestream for time-series event monitoring. AWS Athena sits right on top of the flat storage lake as a serverless query engine, dynamically spinning up on-demand processing nodes to calculate standard SQL requests directly across the files sitting inside the S3 Enriched bucket. An interactive Amazon QuickSight BI Reports dashboard sits at the end of the line, connecting natively to AWS Athena to pull the calculated datasets and render visual analytical insights for business users.

Challenges

Database Strain During Traffic Spikes:

When simulation traffic spiked, my Lambda function tried to write data into the MySQL database row-by-row. This quickly flooded the database connection pool and slowed down performance. I document this in the code and map the production fix.

Synchronizing AWS DMS Configuration:

Getting the live cloud migration tool (AWS DMS) to read changes from my active database without locking tables or interrupting traffic was a major configuration puzzle. Through trial and error, I found the exact database logging parameters needed.
Find the finalized setup in 14_DMS_Data_migration_and_CDC_commands.txt file, for row-based binary logging (binlog_format=ROW) that keeps the data streaming smoothly without any downtime.

Cloud Expense

Traditional EC2 instances and database servers running 24/7 during our initial environment testing drained cloud credits incredibly fast. To fix this, I shifted the data analytics layer to serverless tools (AWS Athena and Kinesis Firehose). This dropped our idle infrastructure costs to zero because project only paid for the exact data it processed.

Strategic Trade off

I trade off sub-millisecond query speeds over AWS Athena to scan flat files directly inside Amazon S3. Because of this architectural choice, my ad-hoc SQL queries take a second or two longer to run. This trade-off reduced cloud infrastructure cost by 70%

To discuss more reach out me on LinkedIn here: https://www.linkedin.com/in/pratik-dhande/

My Github: https://github.com/pratik585/iot-data-migration-lakehouse-aws.git

Here, I create a unified system for all authorities to address water issues effectively. This make collaboration easier and decision making faster and smoother.

Here I created a synthetic water sensor dataset using AWS IoT core. The sensors are supposed to be spread across European countries. Sensors are programmed to behave like recording water quality data in real time. This makes strong and stable framework very necessary. The data comes from many different time zones and countries. It covers various type of water bodies and plants. Such data becomes vital when climates and habitats change.

I used #Microsoft-Azure services. Azure Synapse Analytics is very effective for solving BI problems. It works smooth with Power BI and helps get real-time insights from data. Real-time dashboards are now a standard in the data world.

Raw data starts in MS Excel sheet. The data is moved to an Azure Managed SQL Database using SQL Server Management Studio. The data is moved again to Azure Blob Storage using Azure Logic Apps. I performed batch processing. But we can wire the Logic apps to wait for new changes and only move the new changes forward. Data is stored as JSON objects in the blob storage. Blob storage acts as an intermediate staging area before data is moved down to Azure Data lake Gen2 from final processing. Blob also works as a backup of the raw data in its original extracted state and it can connect to various kind of products to feed the backup data in case of systems failures.

We use Azure Data Factory for moving this huge amount of data from Blob storage to ADLG2. I used a 'copy pipeline' for this task. Once the data reached in Azure Data Lake Gen2, it is ready for use with Azure Databricks.

Further, the data is transformed using Azure Databricks following a Medallion Architecture. This architecture has standard three layers - Bronze, Silver and Gold or data processing. This architecture is very friend for auditing tasks.

Bronze Layer: Raw data is pulled into a Spark data frame
Silver layer: The data is cleaned and refined for quality
Gold Layer: Final processing happens, and data is stored in a table

Data from Gold Layer goes to Power BI and is used to create dashboards. It allows detailed analysis of water quality data

The following columns represent the level of detail or drill-down of parameters at which water readings were taken by the sensors:

View Full Dataset Schema (31 Columns)

Final Analytics

The Ultimate goal of this Medallion Architecture is to transform millions of raw sensor records into a high-performance dashboard. By connecting the Azure Databricks Gold Layer to Power BI , I developed a comprehensive visualization suite to monitor water safety across 39 countries.

Following image shows dashboard that displays Key Environmental Health and Safety Metrics:

This view focuses on the "Safety Levels" and deterministic trends of water quality.

Water Quality Index (WQI): Using custom DAX logic, I calculated a normalized safety level (WQI) by country, providing a clear benchmark against safety thresholds
Temporal Analysis: The "Determinant level across years" chart tracks historical trends, allowing authorities to identify long-term improvements or degradations in water quality
Concentration Treemaps: A visual breakdown of concentration levels by determinant (e.g., Dissolved Oxygen, BOD5, Nitrates) helps quickly pinpoint which chemical factors are most prevalent in a specific region

Following image shows a filter page that serves as a deep-dive tool for the users

Multi-Dimensional Filtering: The dashboard allows for granular filtering by Country, Determinand, and Water Body Category (e.g., Lake Water vs. Ground Water)
Determinand Distribution: Tabular reports show the exact distribution of parameters across different water bodies. This ensures safety measures can be tailored to the specific vegetation and habitat conditions of the region
Scale of Operations: The high-level KPIs highlight the scale of the dataset—managing nearly 600 unique determinands across thousands of monitoring sites

We load only the high quality essential data from the curated Gold Layer into Power BI. The dashboard maintains high performance even when interacting with large historical dataset. This ensures real time insights are available

You can find the complete implementation, Terraform scripts, and Databricks notebooks in my Github

If you enjoyed this deep dive into Azure data engineering, feel free to connect with me on LinkedIn and we can discuss more!