Summary
View the arXiv article here: https://arxiv.org/abs/2305.04411
SmartState was designed to improve the efficiency, quality, and impact of research studies involving human participants. It is a state-based system that acts as a personal agent for each participant, continuously managing and tracking their unique interactions. It consists of four main components: a messaging service, a state machine, a conversational AI, and a web-based platform. Participants primarily engage with the system through a messaging service via texting, emailing, or calling, and can also interact via a chat interface hosted on the web platform.
Key features and benefits of SmartState include:
- Automated management of participant interactions, communications, and data collection, reducing the workload on researchers and preventing errors or inconsistencies.
- Use of finite state machines (FSMs) to track participant progress and ensure adherence to study protocols, preventing critical steps from being skipped. FSMs model system behavior as a series of transitions between well-defined states, where specific conditions must be met for a transition to occur. Once an FSM is created, the paths defined in the graph are fixed and cannot be deviated from.
- Integration of a large language model (LLM) to facilitate natural language interactions with participants. The LLM is used to better understand natural language inputs from participants, providing structured and thematic data. SmartState uses Meta’s Llama 3.1 Instruct model as a base, accessed through a local LLM self-service provider, and can be used with any LLM compatible with OpenAI’s programming interface. The LLM can invoke external tools programmatically to add additional structure and context to conversations and perform tasks beyond language generation. An additional FSM is utilized to handle situations where a response is inadequate for a given question.
- Provision of a web-based platform for researchers to access all functionality, participant logs, reports, and messaging utilities. Beyond storing and displaying responses to questions, the web platform facilitates participant registration, communication, and various analysis functions. Authorized researchers can easily access and download participant metrics, enabling seamless interaction with the underlying system.
- Increased scalability and efficiency for large studies, making it applicable to many research areas. The system overcomes the limitations of traditional rule-based logic by utilizing state-based methods.
- Detailed auditing capabilities to trace a participant’s exact path through a study. Auditing the system in this way prevents disputes between researchers and participants.
- Data security and compliance are addressed by operating the entire system within a NIST SP 800-53 and HIPAA-compliant datacenter. All data is encrypted both in transit and at rest, and access is strictly controlled through the accompanying website where researchers must log in using University credentials.
- SmartState automatically saves its internal state to stable storage every 15 minutes to safeguard against data loss during server and network disruptions.
How SmartState Works:
SmartState follows protocol-driven workflows that dynamically adapt based on user input. At its core, SmartState uses finite state machines (FSMs) to track participants’ progress in a study. FSMs model system behavior as a series of transitions between well-defined states. At each step, specific conditions must be met for a transition to occur, such as the completion of a task or the passage of time, ensuring the system moves logically from one state to the next. Once a study protocol graph is defined, it must be converted into code that captures the state machine structure and conditions under which transitions may occur. This process is automated using a web-based compiler called Umple, which helps generate FSM code and graphs. The paths defined in the graph are fixed and cannot be deviated from, forcing participants to pass through specific checkpoints or achieve a specific status before reaching a goal. FSMs also capture detailed logs of every message received or sent, every state entered, and the transition followed, allowing researchers to trace a participant’s exact path through a study.
SmartState uses a large language model (LLM) to facilitate communication between the system and participants. The incorporation of an LLM enables the system to better understand natural language inputs from participants, providing structured and thematic data. The LLM can use external functions to provide additional context to the conversation. Tool calling allows an LLM to extract arguments for a function to check if a response is adequate. Depending on the output of this function, the model is given additional context to reiterate a question or continue a conversation by storing the response and moving to the next question. An additional FSM is utilized to handle situations where a response is inadequate for a given question. If the response is not satisfactory, context is added, and the question is asked again.
A web-based platform is provided for researchers to access all functionality, participant logs, reports, and messaging utilities. This platform facilitates participant registration, communication, and various analysis functions. Researchers can access and download participant metrics, enabling seamless interaction with the underlying system. The web platform provides user-friendly interactions with the FSM, enabling researchers to add participants, modify study assignments, and monitor both current and past participants’ states. Researchers can also manually transition participants between states if necessary.
Collaborative Projects in Clinical Research using SmartState mentioned in the document include:
- Time-Restricted Eating Intervention in Postmenopausal Women
- Optimization of Preoperative Treatment & Interactive Medical Assistance for Learning in Cardiothoracic Surgery (OptimalCT)
- Plant-based Diet for Diabetes Prevention (PDDP)
Code
SmartState is free and open-source software and available on GitHub here: https://github.com/innovationcore/SmartState-public
For more information on the development of SmartState: An Automated Research Protocol Adherence System read the linked paper.