Landsure Systems (LTSA)
Usability Testing
Building on LTSA's previous study to address concerns with implementations and lingering issues.
Usability Evaluation
Resposibilities
Duration
Tools Used
Team
Study Design, Usability Test Design, Test Facilitation, Data Analysis
Fall 2022 (5 Week Project)
Figma, Google Forms, Zoom
Rowina Chan, Alyssa Lalani, Bowie Rheault
Research + Framing
Project Overview
LandSure Systems Ltd. helps create technological solutions for BC Land Title and Survey Authority (LTSA). LTSA records where land plots are and who owns them.
This is a supplementary, follow-up study for the online “Change of Mailing Address on Title” form.
I was primarily responsible for designing the screening questionnaire, the questions and script of the usability test, organizing with participants, carrying out usability tests, and conducting data analysis.
This is a supplementary, follow-up study for the online “Change of Mailing Address on Title” form.
I was primarily responsible for designing the screening questionnaire, the questions and script of the usability test, organizing with participants, carrying out usability tests, and conducting data analysis.
LandSure is a wholly-owned subsidiary of LTSA
The Problem
We worked with LandSure Systems to create a follow-up usability evaluation of their (then) recently released “Change of Mailing Address on Title” online form. The main objective was to validate, update and go over any gaps in knowledge from their previous study.
Original design of the form
What Are We Looking For?
This evaluation is summative in nature. We designed this study to collect info that would validate past data, and improve LandSure's transition to online forms moving forward.
We sought to examine three main dimensions of usability; Efficiency, Errors, and Satisfaction.
We sought to examine three main dimensions of usability; Efficiency, Errors, and Satisfaction.
Efficiency
Understand the factors that may impact the speed and usability of completing the web form.
Errors
Identify the areas of the workflow and interface that caused errors.
Satisfaction
Discover areas of the web form that may cause frustration, and how to make the process more pleasant.
Designing the Study
Each user test can be broken down into four parts, conceptual model extraction, mid-test questionnaire, direct observation through the think-aloud method, and a post-test questionnaire.
A Conceptual Model was included in this study to collect supplementary insights into how people found the form and the LTSA, which are transferable to LandSure's other online endeavors. Thus collecting insights beneficial beyond the scope of this project.
To validate and refine the Efficiency and reduce Errors caused by the form, receiving immediate commentary as users progress through the form will provide important insight into the exact areas imposing pain points. Thus, the design of the study makes use of the think-aloud method on top of general observation during usability tests.
Additionally, the think-aloud method was chosen as it parallels the previously conducted evaluation by LTSA, so the data produced is comparable to previous results.
A Post-Test System Usability Scale (SUS) questionnaire was used to gather additional data and feedback on their experience and Satisfaction with the web form. The questionnaire included both open and closed-ended questions to reaffirm findings and catch potentially missed insights from the direct observation period.
A Conceptual Model was included in this study to collect supplementary insights into how people found the form and the LTSA, which are transferable to LandSure's other online endeavors. Thus collecting insights beneficial beyond the scope of this project.
To validate and refine the Efficiency and reduce Errors caused by the form, receiving immediate commentary as users progress through the form will provide important insight into the exact areas imposing pain points. Thus, the design of the study makes use of the think-aloud method on top of general observation during usability tests.
Additionally, the think-aloud method was chosen as it parallels the previously conducted evaluation by LTSA, so the data produced is comparable to previous results.
A Post-Test System Usability Scale (SUS) questionnaire was used to gather additional data and feedback on their experience and Satisfaction with the web form. The questionnaire included both open and closed-ended questions to reaffirm findings and catch potentially missed insights from the direct observation period.
Screening Questionnaire
In order to reduce bias, ensure the validity of the study and gaining access to a diverse set of property owners in BC, I created a Google Forms screening questionnaire which eliminated those with a background in legal, land or real-estate fields of work.
The questionnaire was then distributed through a mix of snowball and word-of mouth sampling methods.
The screening process brought in 11 final participants which fit our intended user demographic and responded to the Google form questionnaire, whom I subsequently organized the testing sessions with through email and Google Calander.
The questionnaire was then distributed through a mix of snowball and word-of mouth sampling methods.
The screening process brought in 11 final participants which fit our intended user demographic and responded to the Google form questionnaire, whom I subsequently organized the testing sessions with through email and Google Calander.
Of the 12 respondents we ended with 11 final participants
Designing the Usability Test
I lead the development alongside my fellow test facilitator of a standardized script to keep tests standardized and streamlined.
A set of hypothetical scenario and user data were also created by us to ensure the validity of the study, by preventing participants from disclosing their own information and the reproducibility of the test.
Keeping the Internal Validity of the study in mind, I ensured questions I wrote in the Mid-test Questionnaire were not be compound or leading in any way, whilst still driving the study's investigations into the three main dimensions of usability.
A set of hypothetical scenario and user data were also created by us to ensure the validity of the study, by preventing participants from disclosing their own information and the reproducibility of the test.
Keeping the Internal Validity of the study in mind, I ensured questions I wrote in the Mid-test Questionnaire were not be compound or leading in any way, whilst still driving the study's investigations into the three main dimensions of usability.
A hypothetical user scenario and user
data was provided to participants
data was provided to participants
Fascilitating Tests
I coordinated with qualifying respondents to organize their respective sessions. All hour-long usability tests were conducted remotely over zoom. I facilitated 5 out of 11 usability tests.
Conceptual model extraction
I asked participant to search for pages they think would help accomplish the task at hand.
Participants were given a scenario where they would like to change the recipient of important notices about their property. Participants were then prompted to begin with an empty web browser and asked how they would go about the outlined tasks. This was done before any information about LTSA or the specific online form was revealed to them. The goal was to collect information regarding the discoverability of the web form and insight into the search process.
Mid-test questionnaire
Participants were given a questionnaire to gain further awareness about their prior understanding of LTSA and the “Change of Mailing Address on Title” form.
Direct observation using the think-aloud method
Beyond recording these usability tests, I along with an assistant note-taker made notes as participants engaged with the online form and vocalized their thought processes. Participants were only prompted when they explicitly asked for help, or were visibly stuck to reduce bias.
Post-test questionnaire
Lastly, a post-test System Usability Scale (SUS) questionnaire was administered to gather additional data and feedback on their experience with the web form. The questionnaire included both open and closed-ended questions to reaffirm findings and catch potentially missed insights from the direct observation period.
Conceptual model extraction
I asked participant to search for pages they think would help accomplish the task at hand.
Participants were given a scenario where they would like to change the recipient of important notices about their property. Participants were then prompted to begin with an empty web browser and asked how they would go about the outlined tasks. This was done before any information about LTSA or the specific online form was revealed to them. The goal was to collect information regarding the discoverability of the web form and insight into the search process.
Mid-test questionnaire
Participants were given a questionnaire to gain further awareness about their prior understanding of LTSA and the “Change of Mailing Address on Title” form.
Direct observation using the think-aloud method
Beyond recording these usability tests, I along with an assistant note-taker made notes as participants engaged with the online form and vocalized their thought processes. Participants were only prompted when they explicitly asked for help, or were visibly stuck to reduce bias.
Post-test questionnaire
Lastly, a post-test System Usability Scale (SUS) questionnaire was administered to gather additional data and feedback on their experience with the web form. The questionnaire included both open and closed-ended questions to reaffirm findings and catch potentially missed insights from the direct observation period.
Analysis
Breakdown
The Efficiency and Satisfaction dimensions were both analyzed in similarity, from general observations, comments for each page of the web form and post-test questionnaires with Affinity Diagramming.
For each respective page, I grouped participant-generated sentiments with shared themes together, and highlighted any notes that had a higher-level connection to other pages, then synthesized a larger affinity diagram pertaining to all the sentiments.
The Errors dimension was analysed by documenting points of interest throughout the note-taking and recording review process. Specifically, the number of errors that participants made, if users had to be prompted during the usability test, and if the incorrect information was submitted. This provided us with quantitative metrics that aided our qualitative data interpretations, this allowed us to identify trends or patterns in the data, and provided a more detailed understanding of the user experience.
For each respective page, I grouped participant-generated sentiments with shared themes together, and highlighted any notes that had a higher-level connection to other pages, then synthesized a larger affinity diagram pertaining to all the sentiments.
The Errors dimension was analysed by documenting points of interest throughout the note-taking and recording review process. Specifically, the number of errors that participants made, if users had to be prompted during the usability test, and if the incorrect information was submitted. This provided us with quantitative metrics that aided our qualitative data interpretations, this allowed us to identify trends or patterns in the data, and provided a more detailed understanding of the user experience.
Affinity Diagram synthesized from organised participant sentiments.
Notable Findings
BC land-owners have a lot of pre-conceived notions about their property ownership, and tapping into that can make interface much more intuitive.
Two overarching themes—language and trust—are highlighted by the main findings of this usability test.
The majority of participants were unfamiliar with the LTSA and consequently with a lot of the jargon utilized throughout the form.
According to the findings of our study, the design did not speak the language of its users, violating Nielson Norman's second usability heuristic, "Match Between the System and the Real World." (Nielsen, 2020) This is significant to keep in mind since all British Columbian property owners, regardless of their technical proficiency or familiarity with land titles, should be able to access the form.
Some participants felt a perceived lack of trust; specifically the lack of identity verification, authorization requirements, and legitimacy of the LTSA. However, it is unclear whether this issue was a result of the system, or limitations caused by the study due to the hawthorne effect; participants may have read through copy and instructions quicker than they normally would on their own, which could have also led to the uneasiness they felt.
Two overarching themes—language and trust—are highlighted by the main findings of this usability test.
The majority of participants were unfamiliar with the LTSA and consequently with a lot of the jargon utilized throughout the form.
According to the findings of our study, the design did not speak the language of its users, violating Nielson Norman's second usability heuristic, "Match Between the System and the Real World." (Nielsen, 2020) This is significant to keep in mind since all British Columbian property owners, regardless of their technical proficiency or familiarity with land titles, should be able to access the form.
Some participants felt a perceived lack of trust; specifically the lack of identity verification, authorization requirements, and legitimacy of the LTSA. However, it is unclear whether this issue was a result of the system, or limitations caused by the study due to the hawthorne effect; participants may have read through copy and instructions quicker than they normally would on their own, which could have also led to the uneasiness they felt.
Recommendations
Discoverability
Since the LTSA has relatively low discoverability, many individuals may first turn elsewhere to update their information.
On the LTSA website, only one participant was able to locate the "Change of Mailing Address on Title" form; the majority of participants instead went to BC Assessment, the Government of BC, or the City of Vancouver.
It is advised that the aforementioned websites have a specific section on their individual web pages that not only directs to LTSA but also describes the distinction between their applications.
On the LTSA website, only one participant was able to locate the "Change of Mailing Address on Title" form; the majority of participants instead went to BC Assessment, the Government of BC, or the City of Vancouver.
It is advised that the aforementioned websites have a specific section on their individual web pages that not only directs to LTSA but also describes the distinction between their applications.
Forwarding inquiries from other incorrect sites
Improving PID Discovery
The importance of proximity to relevant form fields was not taken into consideration in either the old or new designs for the Parcel Identifier (PID) help content.
Adding UI elements to increase contextual awareness and improve visibility in the area where users are stuck, the help content should be moved closer to the form field PID entry.
In addition, a number of participants had trouble locating their PID on the BC Assessment website. If LTSA can use its internal registry to build a civic address lookup, it will eliminate the requirement for users to exit the form and better fit with their conceptual model of property.
Currently, the review page just displays the PID and a pre-populated legal description, breaching Nielsen Norman Heuristic #6 Recognition over Recall (Nielsen, 2020) by asking users to memorize their PID from the BC Assessment. Adding the civic address of the matching PID would help users review their information and ensure its correctness.
Adding UI elements to increase contextual awareness and improve visibility in the area where users are stuck, the help content should be moved closer to the form field PID entry.
In addition, a number of participants had trouble locating their PID on the BC Assessment website. If LTSA can use its internal registry to build a civic address lookup, it will eliminate the requirement for users to exit the form and better fit with their conceptual model of property.
Currently, the review page just displays the PID and a pre-populated legal description, breaching Nielsen Norman Heuristic #6 Recognition over Recall (Nielsen, 2020) by asking users to memorize their PID from the BC Assessment. Adding the civic address of the matching PID would help users review their information and ensure its correctness.
Recommended UI redesign for locating PID
Conclusion
Moving forward, a field evaluation can be carried out with a range of participant ages to get a more accurate understanding of the application's strengths and disadvantages. In a field evaluation, users' actual data from the form would be used to monitor how they interact with the interface, which is a more accurate representation than conforming to a fictional situation.
Additionally, a sample with a wider range of technical competence would provide stronger external validity, making it more accurately reflect and generalized to the broader user population.
Additionally, a sample with a wider range of technical competence would provide stronger external validity, making it more accurately reflect and generalized to the broader user population.