ActivAnnot: Annotation of Physical Sensing Data for Human Activities
This project explores multimodal sensing's role in context-aware services, highlighting the opportunities that can be obtained by exploiting the interplay of different modalities in a smart environment. It develops an ML-based adversarial framework for mobility inference using keystroke patterns, an automated annotation system for human activity recognition (HAR), further extends it for a dual-user environment, and finally performs a detailed analysis of the micro-activities in complex activities of daily living (ADL) from the purview of label informativeness.
Key Features
Uniqueness: The works associated with the thesis allow us to annotate raw sensor streams from inertial sensors using environmental audio data.
- Multimodal: The frameworks developed as parts of the study are multimodal and consider inertial data and acoustic data as key inputs.
- Automated Annotation: The key objective of these frameworks is to exploit the interplay of multiple modalities present in a smart environment for automated annotation.
- Reduction of Human Effort: The key advantage offered by these frameworks is the reduction of overall human effort and cost associated with labeling sensor data.
- Human annotations: Real-time activity labels collected via a speech-to-text app, providing necessary context for interpreting pollution readings.
- Real-world Scenarios: All the frameworks are tested on real-world scenarios, with some involving more than one user performing complex human activities of daily living.
- Deeper Insights on Annotation: Finally, all these works present a detailed investigation of how human annotations can be challenging and how different modalities can assist each other in mitigating these challenges.
ConType (PerCom'20)
LASO (ICMI'20)
AmicroN (SmartComp'22)
Acconotate (DCOSS-IoT'23)
GitHub (Acconotate)
Contributors
Soumyajit Chatterjee
Nokia Bell Labs, UK
Bivas Mitra
IIT Kharagpur, India
Sandip Chakraborty
IIT Kharagpur, India
Teaser Video
Publications
- Soumyajit Chatterjee, Soumyajit, Bivas Mitra, and Sandip Chakraborty. "Type2motion: Detecting mobility context from smartphone typing." In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pp. 753-755. 2018.
- Soumyajit Chatterjee, Adrija Bhowmik, Arun Singh, Surjya Ghosh, Bivas Mitra, and Sandip Chakraborty. "Detecting Mobility Context over Smartphones using Typing and Smartphone Engagement Patterns." In 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 1-8. IEEE, 2020.
- Soumyajit Chatterjee, Avijoy Chakma, Aryya Gangopadhyay, Nirmalya Roy, Bivas Mitra, and Sandip Chakraborty. "LASO: Exploiting locomotive and acoustic signatures over the edge to annotate IMU data for human activity recognition." In Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 333-342. 2020.
- Soumyajit Chatterjee, Bivas Mitra, and Sandip Chakraborty. "Non-intrusive Continuous User Identification from Activity Acoustic Signatures." In 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), pp. 356-359. IEEE, 2021.
- Soumyajit Chatterjee, Bivas Mitra, and Sandip Chakraborty. "AmicroN: Framework for Generating Micro-Activity Annotations for Human Activity Recognition." In 2022 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 26-31. IEEE, 2022.
- Soumyajit Chatterjee, Bivas Mitra, and Sandip Chakraborty. "Automated Micro-Activity Annotations for Human Activity Recognition with Inertial Sensing." In 2022 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 162-164. IEEE, 2022.
- Soumyajit Chatterjee, Arun Singh, Bivas Mitra, and Sandip Chakraborty. "Acconotate: Exploiting Acoustic Changes for Automatic Annotation of Inertial Data at the Source." In 2023 19th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), pp. 25-33. IEEE, 2023.