Project background

The AR/XR Orthopedic Doctor’s Assistant is a widget or mini application designed to enhance the work of orthopedic doctors by utilizing augmented reality (AR) or mixed reality (XR) technology. Its primary goal is to streamline and facilitate the daily tasks and responsibilities of orthopedic healthcare professionals.

The application provides doctors with real-time information and visualizations, allowing them to efficiently track patient queues, access detailed patient profiles, and navigate through their workflow seamlessly. By leveraging AR/XR capabilities, doctors can view the skeletal structure of each patient, with clear markings indicating fractures and problem areas. This visual representation aids in accurate diagnosis and treatment planning.

Furthermore, the widget incorporates a comprehensive digital journaling system that enables doctors to maintain detailed records of patient interactions, medical observations, and treatment plans. The assistant leverages artificial intelligence (AI) to analyze and interpret the patient data, providing intelligent insights and recommendations for the next steps in patient care.

The AR/XR Orthopedic Doctor’s Assistant aims to optimize orthopedic healthcare workflows by merging the power of advanced visualization, patient management, and AI-driven assistance. It empowers orthopedic doctors to make informed decisions, streamline their processes, and ultimately improve patient outcomes.

My role

My role in the project is focused on research, UX/UI design, 3D design, VR/AR design, and prototyping. I am responsible for conducting thorough research to understand the needs and challenges of orthopedic doctors, as well as the expectations of patients. This research helps inform the design process and ensures that the AR/XR Orthopedic Doctor’s Assistant meets the specific requirements of the target users.

In terms of UX/UI design, I am responsible for creating intuitive and user-friendly interfaces for the application. I work closely with the development team to ensure that the interface design aligns with the functionality and capabilities of the AR/XR technology. I pay attention to the layout, visual hierarchy, and navigation, aiming to provide a seamless and efficient user experience for orthopedic doctors.

Additionally, I contribute to the 3D design aspect of the project. The 3D models are crucial in providing doctors with a visual representation of fractures and problem areas, aiding in accurate diagnosis and treatment planning.

Furthermore, my expertise extends to VR/AR design, where I leverage the capabilities of these technologies to create immersive and realistic experiences for the orthopedic doctors. This involves designing interactive virtual or augmented environments where doctors can visualize patient data, access real-time information, and perform various tasks efficiently.

Lastly, I take part in prototyping, creating interactive and functional prototypes that simulate the core features and functionalities of the AR/XR Orthopedic Doctor’s Assistant. These prototypes help validate design concepts, gather feedback from stakeholders, and refine the overall user experience before moving into the development phase.

The problem

Despite advancements in orthopedic healthcare, many doctors still face challenges in efficiently managing their daily tasks and visualizing patient conditions. The traditional methods of patient management and diagnosis often result in time-consuming processes and limited visualizations, leading to potential inefficiencies and compromised patient outcomes.

Manual Patient Tracking: Doctors often rely on manual methods, such as paper-based records or basic digital systems, to track patient queues and access patient profiles. This manual process can be prone to errors, delays, and difficulties in retrieving relevant information quickly. As a result, doctors may struggle to prioritize patients effectively and provide timely care.

Limited Visualization Tools: Conventional diagnostic tools, such as X-rays and MRI scans, provide valuable insights into patient conditions. However, they often lack detailed visualizations that allow doctors to precisely assess fractures, problem areas, and other critical aspects. This limitation can lead to misinterpretations, inaccurate diagnoses, and suboptimal treatment planning.

Fragmented Data Management: With the increasing volume of patient data, orthopedic doctors face challenges in organizing and accessing comprehensive patient records. The absence of a unified digital journaling system makes it difficult to track medical observations, treatment plans, and follow-up appointments effectively. This fragmentation of data can hinder the holistic understanding of patients’ conditions and impede collaborative decision-making among healthcare professionals.

Lack of Intelligent Insights: The absence of artificial intelligence (AI)-driven assistance further exacerbates the challenges faced by orthopedic doctors. Without intelligent insights and recommendations based on patient data analysis, doctors may struggle to leverage data effectively and make informed decisions regarding treatment options, potential risks, or recovery timelines.

The goal

The goal of the AR/XR Orthopedic Doctor’s Assistant project is to optimize orthopedic healthcare workflows and enhance patient care by leveraging augmented reality (AR) or mixed reality (XR) technology. The project aims to achieve the following objectives:

Streamline Workflow Efficiency: The project seeks to streamline the daily tasks and responsibilities of orthopedic doctors by providing them with a comprehensive digital assistant. The assistant will enable doctors to efficiently track patient queues, access detailed patient profiles, and navigate through their workflow seamlessly. By automating manual processes and centralizing patient information, the goal is to reduce administrative burdens, save time, and improve overall workflow efficiency.

Enhance Visualization and Diagnosis: By leveraging AR/XR capabilities, the project aims to enhance the visualization of patient conditions for orthopedic doctors. The assistant will provide real-time, interactive visualizations of skeletal structures with clear markings indicating fractures and problem areas. This advanced visualization will aid doctors in accurate diagnosis, treatment planning, and surgical procedures, ultimately improving patient outcomes.

Enable Comprehensive Data Management: The project aims to address the challenge of fragmented data management by incorporating a comprehensive digital journaling system. The assistant will allow doctors to maintain detailed records of patient interactions, medical observations, and treatment plans in a unified and easily accessible format. This centralized data management system will enhance data organization, facilitate collaborative decision-making, and promote continuity of care.

Leverage AI for Intelligent Insights: The AR/XR Orthopedic Doctor’s Assistant project intends to leverage artificial intelligence (AI) capabilities to analyze and interpret patient data. By employing AI algorithms, the assistant will provide intelligent insights and recommendations based on patient information. These insights can help doctors in making informed decisions regarding treatment options, potential risks, and personalized care plans.

Improve Patient Outcomes: The ultimate goal of the project is to improve patient outcomes in orthopedic healthcare. By streamlining workflows, enhancing visualization, and providing intelligent insights, the assistant aims to empower orthopedic doctors to deliver more accurate diagnoses, efficient treatments, and personalized care. The project strives to enhance the overall patient experience, minimize complications, and promote successful recovery and rehabilitation.

Project duration

April 2023 to June 2023.

Meet our Personas

The Research

As a researcher for the AR/XR Orthopedic Doctor’s Assistant project, my approach involves gathering insights through personal interviews, leveraging Google search and existing resources, utilizing ChatGPT for knowledge expansion, and conducting usability tests.

Through personal interviews, I engage with orthopedic doctors, physical therapists, and clinic administrators to understand their workflows, pain points, and expectations regarding the assistant. These interviews provide valuable qualitative data, offering in-depth insights into their needs and preferences.

To enhance my understanding, I conduct extensive research using Google search and review existing resources such as academic papers and industry reports. This helps me gather insights on emerging trends, AR/XR applications in healthcare, and usability design principles.

Additionally, I leverage ChatGPT, an AI language model, to expand my knowledge and gain insights on relevant topics. ChatGPT assists me in answering specific questions, generating ideas, and providing additional context for the research.

To evaluate usability, I design and conduct usability tests with representative users. Through interactive prototypes of the assistant, I observe how users interact with the interface and collect feedback on usability, navigation, visualizations, and overall user experience. This feedback helps identify areas for improvement and refine the design.

Analyzing and synthesizing the data gathered from personal interviews, Google search, ChatGPT interactions, and usability tests allows me to identify common themes, patterns, and user preferences. These findings provide a comprehensive understanding of user needs, challenges, and expectations.

Using the research insights, I iterate and refine the design of the AR/XR Orthopedic Doctor’s Assistant. Incorporating user feedback and addressing usability issues, I aim to create a user-centered solution that meets the specific needs of orthopedic healthcare professionals.

Through this research process, I strive to gather valuable insights, validate design decisions, and ensure the development of an efficient and user-friendly AR/XR Orthopedic Doctor’s Assistant.

The most important pain points

Inefficient Workflow: Orthopedic doctors often face challenges in managing their workflow efficiently. This includes tasks such as patient tracking, accessing patient information, and navigating through their daily responsibilities. The lack of streamlined processes and digital tools can lead to time-consuming administrative tasks and delays in patient care.

Limited Visualization Capabilities: Traditional diagnostic tools may have limitations when it comes to visualizing the skeletal structure and identifying fractures or problem areas accurately. Orthopedic doctors rely heavily on visual information to make accurate diagnoses and plan effective treatments. Inadequate visualization capabilities can hinder their ability to provide optimal care to patients.

Fragmented Data Management: Current data management practices in orthopedic healthcare settings can be fragmented, with patient information spread across multiple systems or in paper-based formats. This makes it challenging for doctors to access comprehensive patient profiles, leading to potential errors, delays, and difficulties in maintaining detailed records of interactions and treatment plans.

Lack of Intelligent Insights: Orthopedic doctors would benefit from intelligent insights and recommendations based on patient data. The absence of AI-driven assistance can limit their ability to analyze and interpret complex patient information, potentially resulting in missed opportunities for personalized care and treatment optimization.

User Experience and Adoption: The success of the AR/XR Orthopedic Doctor’s Assistant relies on user adoption and a positive user experience. If the assistant is not intuitive, user-friendly, or does not align with the existing workflows and preferences of orthopedic doctors, it may face resistance or low adoption rates, hindering its potential to improve patient outcomes.

My problem statements

Emily (user) is an orthopedic surgeon (user characteristics), who needs a solution to streamline her workflow (user need) because this will improve the patient care (insight).

Sarah (user) is a physical therapist (user characteristics), who needs a comprehensive view of patients’ medical history and treatment plans to provide personalized care effectively (user need) because this will result seamless collaboration with doctors to optimize patient outcomes (insight).

Mark (user) is an Orthopedic Clinic Administrator (user characteristics), who needs an easier way of patient management (user need) because this will allow him to have more comprehensive patient records and collaborate effectively with doctors and other healthcare professionals (insight).

Usability study

Study type: Remote Usability Study

Location: Conducted online using video conferencing and screen sharing tools.

Participants: A diverse group of orthopedic doctors and physical therapists with varying levels of experience and familiarity with AR/XR technology. Aim for a sample size of 8-10 participants.

Duration: The usability study is estimated to last approximately 60-90 minutes per participant, including setup, tasks, and post-task interviews.

Results: The usability study aims to identify usability issues, gather user feedback, and obtain valuable insights to refine the design of the AR/XR Orthopedic Doctor’s Assistant, ultimately improving its usability and user experience for orthopedic doctors and physical therapists.

VR Concept

Low-fidelity design (wireframes)

Greyboxing (ShapesXR)

High-fidelity designs (ShapesXR and Bezel)


Reasoning behind decisions

The decision to use blue as the primary color in the interface design of the AR/XR Orthopedic Doctor’s Assistant was based on its medical association, calming effect, positive symbolism, and visual contrast. By incorporating blue, the application creates a visually appealing and user-friendly environment that aligns with the needs and expectations of orthopedic doctors and healthcare professionals.

The decision to utilize augmented reality (AR) and mixed reality (XR) technology in the AR/XR Orthopedic Doctor’s Assistant was driven by several key factors:

Enhanced Visualization: AR and XR technology provide immersive and interactive experiences that enhance visualization capabilities. By overlaying digital information onto the real world or merging virtual and real elements, orthopedic doctors can visualize and interact with patient data, skeletal structures, and medical imagery in a more intuitive and informative manner.

Real-time Information: AR and XR technology enable real-time data integration, allowing orthopedic doctors to access up-to-date patient information, medical records, and diagnostic imaging seamlessly. This real-time aspect enhances the efficiency and accuracy of decision-making processes, leading to improved patient care and treatment outcomes.

Hands-Free Interaction: AR and XR headsets offer hands-free interaction, freeing orthopedic doctors from relying solely on traditional input devices. This allows them to manipulate and interact with the virtual content while keeping their hands focused on patient examinations or procedures, improving workflow efficiency and reducing physical constraints.

AR Headset Choice:

The decision to use an AR headset for the AR/XR Orthopedic Doctor’s Assistant was based on the following considerations:

Enhanced Visualization: AR headsets provide a more immersive and accurate visualization of digital information by overlaying it onto the user’s real-world environment. This ensures that orthopedic doctors can view the skeletal structures and medical data in a more realistic and contextually relevant manner, aiding in accurate diagnosis and treatment planning.

Mobility and Flexibility: AR headsets offer the advantage of being lightweight and portable, allowing orthopedic doctors to use them in various clinical settings or while on the move. This mobility facilitates the integration of the assistant into the doctor’s daily workflow, ensuring accessibility and convenience.

3D Buttons:

The decision to utilize 3D buttons instead of 2D buttons in the AR/XR Orthopedic Doctor’s Assistant interface was driven by the following reasons:

Enhanced User Interaction: 3D buttons provide a more intuitive and immersive user interaction experience compared to 2D buttons. They offer a sense of depth and physicality, enabling orthopedic doctors to interact with the interface elements in a more natural and tactile way.

Visual Feedback: 3D buttons can provide visual cues and feedback, such as highlighting or animation, when interacted with, improving the user’s understanding and confirmation of their actions. This feedback mechanism enhances the usability and reduces the likelihood of errors or misinterpretation.

Consistency with Spatial Environment: In an AR or XR environment, where the virtual and real worlds coexist, using 3D buttons aligns with the spatial context and creates a more cohesive and immersive user experience. It helps bridge the gap between the physical and digital elements, resulting in a more seamless and engaging interaction.

Overall, the decisions to incorporate AR and XR technology, utilize AR headsets, and implement 3D buttons in the AR/XR Orthopedic Doctor’s Assistant were made to enhance visualization, provide real-time information, offer hands-free interaction, ensure mobility, improve user interaction, and maintain consistency with the spatial environment. These decisions collectively contribute to the goal of streamlining workflows and improving the overall user experience for orthopedic doctors.

Accessibility considerations

Ensuring accessibility in the design of the AR/XR Orthopedic Doctor’s Assistant is crucial to accommodate users with diverse needs and provide an inclusive experience. Here are some key accessibility considerations for the project:

Visual Accessibility:
Provide adjustable font sizes and high contrast options to accommodate users with visual impairments.
Ensure sufficient color contrast between text and background elements to enhance readability.
Incorporate alternative text descriptions for visual content, allowing screen readers to provide audio descriptions.

Motor Accessibility:
Design user interfaces with large interactive elements, such as buttons and icons, to facilitate easy selection for users with motor impairments.
Implement support for various input methods, including touch gestures, voice commands, and keyboard shortcuts, to accommodate users with different motor capabilities.

Hearing Accessibility:
Provide visual cues and feedback in addition to audio alerts or notifications to ensure important information is accessible to users with hearing impairments.
Include closed captioning or subtitles for any audio or video content used within the application.

Cognitive Accessibility:
Keep the user interface simple and intuitive, minimizing cognitive load and avoiding complex or overwhelming visual designs.
Provide clear and concise instructions, tooltips, and help resources to assist users in understanding the functionality of the application.
Assistive Technology

Ensure compatibility with screen readers, screen magnifiers, and other assistive technologies commonly used by individuals with disabilities.
Conduct compatibility testing with assistive technology tools to verify the application’s usability and functionality for users relying on such tools.

User Feedback and Testing:
Involve users with disabilities in the usability testing phase to gather their feedback and identify any accessibility barriers or challenges.
Actively incorporate user feedback to address accessibility issues and make necessary improvements throughout the development process.
By incorporating these accessibility considerations, the AR/XR Orthopedic Doctor’s Assistant can provide a more inclusive and accessible experience for all users, regardless of their abilities or disabilities.


The development of the AR/XR Orthopedic Doctor’s Assistant has successfully brought together advanced visualization, patient management, and AI-driven assistance to streamline and enhance the workflow of orthopedic doctors. By leveraging augmented reality (AR) and mixed reality (XR) technology, the application provides real-time information, accurate visualization of skeletal structures, and intelligent insights for improved diagnosis and treatment planning.

Throughout the project, extensive research, including personal interviews, Google searches, ChatGPT interactions, and usability tests, has been conducted to ensure the application meets the needs and preferences of orthopedic healthcare professionals. The feedback and insights gathered from these research activities have informed key design decisions, resulting in a user-centric and intuitive interface.

The project has introduced three personas representing different user profiles, allowing for a deeper understanding of the target users and their goals, frustrations, and motivations. These personas have guided the design process, ensuring the application addresses the specific needs and pain points of orthopedic doctors and healthcare professionals.

Considerations for accessibility have been integrated into the design, ensuring the application is inclusive and accessible to users with diverse needs. By prioritizing visual, motor, hearing, and cognitive accessibility, the AR/XR Orthopedic Doctor’s Assistant aims to provide an inclusive user experience that accommodates individuals with disabilities.

The project has successfully achieved its goal of optimizing orthopedic healthcare workflows by empowering orthopedic doctors with enhanced visualization, efficient patient management, and AI-driven assistance. By combining advanced technology, ergonomic design, and user-centric features, the AR/XR Orthopedic Doctor’s Assistant aims to improve patient outcomes, streamline processes, and support informed decision-making in orthopedic healthcare settings.

In conclusion, the AR/XR Orthopedic Doctor’s Assistant project represents a significant advancement in orthopedic healthcare technology, catering to the unique needs and challenges faced by orthopedic doctors. The application’s ability to leverage AR/XR technology, provide accurate visualization, and offer intelligent assistance holds great promise for transforming the way orthopedic healthcare professionals diagnose, treat, and care for their patients.

Tools used


Try it in VR