In this project, I developed a data lakehouse solution for the STEDI Human Balance Analytics platform using Apache Spark and AWS Glue. The objective is to process data from motion sensors, mobile accelerometers, and customer records, then curate trusted datasets that can support machine learning models for real-time step detection.
The workflow begins by extracting and processing raw data from multiple sources, organizing it into structured layers within a scalable cloud architecture. Using Spark and AWS Glue jobs, I transform and refine sensor data while applying privacy-focused filtering to ensure only consented customer data is used in model training.
As part of the curation process, I build additional transformation jobs to generate step-trainer datasets enriched for machine-learning use cases. This includes joining sensor and accelerometer streams, validating data quality, and preparing optimized datasets for downstream analytics and model development.
Beyond the pipeline itself, the project explores core data engineering concepts such as lakehouse design, distributed processing, privacy-aware data modeling, and scalable cloud-based ETL. The final solution enables data scientists to access reliable training data while supporting the broader goal of accurate real-time balance and step detection.
