Human Activity Recognition (HAR) is the process of identifying and monitoring physical activities through wearable sensor data, providing real-time insights that enhance patient care, personalized fitness planning, and lifestyle interventions. With the growing adoption of wearable devices, demand for reliable HAR systems capable of operating in diverse, free-living environments has increased, as these systems must handle varied data quality. Wearable device-based HAR, leveraging multimodal data, holds significant promise for applications like health monitoring, fall detection, and rehabilitation, where precise, continuous activity tracking is critical.

This study evaluates HAR performance using multimodal data from two wearable devices: the Cosinuss° C-Med° Alpha (an in-ear device) and the Garmin Venu 2 (a wrist-worn smartwatch). Data were collected from eight participants in a free-living setting, with Cosinuss° capturing continuous physiological metrics, including heart rate, blood oxygen saturation, body temperature, and accelerometer data. Garmin also recorded physiological data, such as heart rate and oxygen saturation, in addition to an extensive motion dataset comprising accelerometer, gyroscope, and GPS altitude readings. By independently assessing motion-only and motion plus physiological data, this study investigates how multimodal data integration, specifically adding physiological signals, impacts model performance, particularly for complex activities.

Various machine learning models (KNN, SVM, Random Forest, XGBoost) with handcrafted features and deep learning models (LSTM, ConvLSTM, Transformer) using raw features were evaluated. Machine learning models performed well with motion-only data for both simple and complex tasks, while deep learning models showed notable gains with the addition of physiological data. XGBoost achieved the highest accuracy among machine learning models, while Transformer led in deep learning across single and multi-subject datasets. The LSTM model showed a 30\% improvement in weighted F1 score when using both motion and physiological data from Cosinuss°. Zero shot learning revealed that Cosinuss° performed better than Garmin in motion-only classification, while Garmin’s combined motion and physiological data exhibited superior cross-subject generalization.

The results highlight the value of multimodal data integration in enhancing HAR for complex activities and suggest that Transformer and XGBoost models are particularly suited for robust healthcare and fitness applications. Future work may explore transfer learning and cross-subject training to broaden HAR’s adaptability and generalizability.