This document was created to help OFA staff assess whether the product development team performs satisfactorily and identify areas for improvement. We built this from 18F's evaluation practices for our own internal and external work.
It is not meant to replace the Quality Assurance Surveillance Plan (QASP), acceptance criteria, or to be an exhaustive checklist for all work. These heuristics should compliment contractual evaluative measures and be used to identify feedback opportunities within the team.
Understanding code quality and maintainability requires examining the code itself and comprehending it at least at a high level. The indicators below are generally intended for someone who already has the knowledge to do that, but each one includes steps that a non-coder can take to help determine if things are going well or not.
| Indicator | Good sign | Bad sign |
|---|---|---|
| Code actually does what the stories intend | Incremental code delivery can be traced back to user story value or user story tasks.
Occasionally, there will be concerted efforts to remediate technical debt but this should not be the norm. |
Code does not address users stories or user story acceptance criteria. |
| Code uses Do Not Repeat Yourself (DRY) principles | Common functionality uses open source libraries when possible.
When new code is developed repeat code is centralized into a module. Developers leverage USWDS code and guidance as much as possible, rather than creating custom components. Non dev: The dev team will not have similar code throughout the code base, but will design it so that central modules can be reused. |
Common functionality uses custom code.
The same methods are repeated throughout different files. Custom components that deviate from USWDS guidance are common. |
| Code methods are short and simple, and maintainable | Methods are generally shorter than 25 lines of code. The methods generally have few conditionals to minimize cyclomatic complexity. The team may include automated tests like flake8 to check for code readability and complexity.
Methods are easy to test. Method function can outputs can be reasoned out by review of the method and the accompanying tests. Non dev: Coded methods are short. At the individual-method level, there are only a few pieces of logic. We favor coding styles that allow for testing and clarity. |
Methods are very long and cram a lot of different iterations or conditionals into a single method. Methods are hard to test.
Methods are hard to reason about and thus also difficult to test, difficult to maintain, and prone to bugs. |
| Code has logical organization. | When files become large they are broken out into a folder, and folder structure could be reasonably understood by a developer less-familiar with the project.
Non dev: The code is well organized by file and folder. |
|
| Code variables and methods have human readable names | Method names should accurately reflect what the method does, and variable names should clearly indicate the data they’re holding and for what purpose. Don’t be afraid of long names. Good naming practices contribute to self-documenting code and reduce the manual documentation burden.
Non dev: Naming conventions within the code favor comprehension. |
Variable names and method names are shortened, concatenations, or single letters or very generic and do not give much insight into the function of code. |
| Code is well-tested to prevent regressions | All tests pass on every pull request. When new features are added or bugs are fixed, tests are updated or added as appropriate. Tests meaningfully hit most if not all of the logic of the additional code.
Non dev: Have the dev team create an integration with Github that includes test results with each code change. Make sure the tests pass before approving, and also ask the developers to show you any new or modified tests to give you confidence that tests have been updated as needed. |
Tests are removed, or new tests are not added. Tests are written such that they pass without actually testing anything. The percentage of code tested drops substantially. |
| Code conforms to a single style standard | A linter and/or code formatter are applied to the code to identify and fix inconsistencies before code is accepted. Linting styles are separate for the frontend and the backend.
Non dev: Have the dev team create an integration with Github that includes a linter report with each code change. If the linter report shows no errors, the code is satisfying the style standards. |
Linter errors are ignored and/or linting ignore statements are frequently added to pieces of code. |
| Code is free of known vulnerabilities | Third-party dependencies are routinely checked for known-vulnerable versions. The app as a whole is analyzed by an automated security tool (e.g., Dependabot and OWASP) for common kinds of vulnerabilities.
When vulnerabilities are found, the developer team takes time to investigate the impact and remediate or mitigate the vulnerability. Non dev: Have the dev team create an integration with Github or MSTeams that reports whenever there are vulnerabilities. The developers should be able to discuss with you what those are and how they are being mitigated. |
Vulnerable dependencies are ignored without good reason (e.g., ignoring a vulnerability in a development-only dependency may be fine). Ignoring reports from security scanning tools. |
| Code for deployment is checked in | Any configuration changes are included and support a continuous delivery pipeline to our development environment. | Configurations that were manually added are not documented. |
| Code is well-documented | In places where the code is complex or not sufficiently self-documenting, inline comments are provided to explain it to future developers.
Methods use a standard notation in-line notation that can be parsed. For this project (JSDoc on the frontend) and (Python Docstring on the Backend) External APIs are documented. A wiki or other documentation captures larger system documentation, such as system diagrams, etc. Documentation on how to run the application locally is up to date and can be run through. Non dev: Ask the developers to explain the code’s behavior. If they cannot give a coherent, layperson’s explanation of the behavior of the code, that suggests that they don’t have a deep understanding of it which might further indicate that the code is unnecessarily complex. |
No inline comments because code is “completely self-documenting.”
Documentation is not updated as code changes. Code is commented out instead of removed. |
Design touches every area of the product, and can mean different things to different teams at different times. The work you'll evaluate may vary from sprint-to-sprint; you may review research plans on week and user interface mockups the next. No matter the work, these design heuristics are the specific markers to look for to ensure design decisions are user centered, ethical, and maintainable.
These indicators are meant to help evaluate the practices and work of a design team. In addition to general design heuristics, there are also specific indicators for evaluating accessibility, usability tests and research, and content.
| Indicator | Good sign | Bad sign |
|---|---|---|
| Design process | ||
| Needs are clear | The design team can clearly state what problem a design is trying to solve and what users need from that design.
Understanding of those needs is clearly tied to research. |
It’s not clear what problem a design is trying to solve or how it relates to user needs.
Design choices are based on personal preferences rather than research. |
| Open about uncertainties | The design team is open about where questions exist and actively tries to answer them. They regularly initiate conversations and ask questions. | The design team doesn’t note where uncertainties exist. They passively accept information rather than actively pursue it. |
| Design with, rather than for | The design team involves others in the design process (users, stakeholders, other team members) and uses others’ expertise to shape the design.
The process for how a design was created is clear to the team. Feedback is solicited regularly to check for assumptions and alignment. Time for feedback is built into the design process and adjustments are made to designs after feedback is discussed. |
Design exercises involve only designers. The designer is seen as someone who can make magic in a vacuum and deliver on their own.
Designs come from a “black box” or unclear process. There are limited or no feedback opportunities, and designs feel misaligned with product goals or user needs. Feedback is only requested at the end of the design process or not considered when iterating on the design. |
| Implementation | ||
| Uses reusable design patterns | The design team creates and leverages reusable UI patterns to create consistent designs. They follow USWDS usability and accessibility best practices.
If a deviation is made from an established pattern, the reason is clearly stated, and deviations still feel consistent with the team's prior design choices. |
Designers deviate frequently from past design patterns and create new ones that are not consistent with prior design choices.
Design prioritizes novelty and bespoke solutions over established patterns. |
| Designers and developers collaborate | Designers and developers pair regularly. They work together to identify design approaches that are technically feasible to implement that meet user needs.
Designers and developers give each other feedback frequently, without prompting from other team members, and in a productive, collaborative manner. |
There is frequently misalignment between what is designed and what is implemented.
Designers and developers are isolated from each other, and don’t provide feedback to one another. |
| Integrates content, accessibility, and usability testing heuristics in work | Reference back to content, accessibility, and usability testing heuristics. | Does not follow content, accessibility, and usability testing heuristics |
The following indicators support design of barrier-free digital products that can be used by everyone. The "good signs" and "bad signs" are merely examples; they are in no way comprehensive.
The Principles of Universal Design were initially developed in 1997 at the Center for Universal Design at North Carolina State University, funded by the U.S. Department of Education's National Institute on Disability and Rehabilitation Research.
| Indicator | Good sign | Bad sign |
|---|---|---|
| Equitable use | Everyone uses the same website. | People with disabilities use a separate “accessible” site or an “accessibility mode.” |
| Flexibility in use | People can operate the website even when it is zoomed in many times. | People can’t use a screen reader to identify the headings on the page. |
| Simple and intuitive use | The website is straightforward, with clean layout, consistent interaction, and clear information. | The website is arranged based on the agency’s organizational structure rather than on visitors’ goals, making information difficult to find. |
| Tolerance for error | Interactions are designed to promote success and minimize risks, for example, by providing confirmations and feedback. | People can’t change a response on a previous step, such as when they cancelling or deleting action |
| Low physical effort | Each feature is manually tested for accessibility to ensure that people can efficiently operate websites using their preferred input device (such as a keyboard). | The team does not demonstrate a process for manual accessibility testing.
Website uses insufficient color contrast and small text sizes, causing eye strain. |
The following indicators can help determine if a usability test will produce useful results. This is not an exhaustive list, but it should be helpful in planning and assessing usability tests.
| Indicator | Good sign | Bad sign |
|---|---|---|
| Study design | ||
| Study purpose | Researcher clearly articulates the purpose of the study (for example, as a specific question to answer, an area of inquiry, etc.). | Researcher does not specify a purpose for the study , or the purpose specified is very broad, like “testing the app” or “finding problems.” |
| Number of participants | Studies include a sufficient number of participants necessary to see patterns, and there’s only one participant per session. | Study includes only one participant, or includes multiple participants tested simultaneously where participants can influence one another (like a focus group ). |
| Sampling bias | Participants are a diverse group of people who actually use the application or do tasks that are being investigated. | Participants are experts (vs. average users), friends, family, or close colleagues of the product team. Participants are homogenous with regard to abilities, background, and demographics, and as a result do not represent the user population. |
| Privacy awareness | Researchers get informed consent from each participant before starting research activities so that participants know what the study entails, how their feedback will be used, and their right to opt out.
The research team is aligned on how to hold up their end of informed consent agreements in their research work. (e.g. where records are stored, anonymizing notes) |
Informed consent is not collected and participants are unclear how their data will be used.
The team isn’t clear or does not respect how the informed consent agreement impacts their research work. |
| Moderator style | ||
| Participant comfort | Participants feel comfortable taking some time to give an answer. | Researcher fills any “awkward pauses” with rephrasing or leading questions. |
| Question style | Researcher asks follow up questions that are open-ended and non-leading.
Researchers give participants opportunities to question design decisions and give participants space to share their thoughts and experiences. |
Questions are leading or subtly suggest potential solutions or hypotheses.
Researcher asks participants to confirm design decisions. For example, “This content makes sense, right?” |
| Task variability | Researcher asks participants to complete similar tasks related to the study’s purpose. | Researcher asks participants to complete different tasks, or tasks unrelated to the study’s purpose. |
| Framing | Researcher asks participants to complete tasks with the product or service that align with the participant’s work-related goals. | Researcher asks participants to complete tasks unrelated to their work-related goals. For example, asking a participant how they might send a fax when their job doesn’t call for that. |
| Priming | Researcher asks participants to complete tasks without indicating how that might be done. For example, “How would you view the status of your application?”
Questions allow for multiple answers. For example, “Walk me through what happens when you get a ticket.” |
Researcher guides participants in completing tasks or two a particular answer. For example, “Which of the links in the header would you click to log in?” |
| Team participation | ||
| Number of teammates | The team designates a single moderator for the test, and at least one member of the product team observes the usability test.
researcher is mindful of who’s participating in a research session and limits observers so that participants feel comfortable sharing |
A single person from the product team participates in and leads the study.
include a large number of observers that make participants uncomfortable (e.g. more than four non participants). |
| Observer role | Observers do not moderate. They are generally quiet, and ask open-ended questions after the session has concluded. | Observers interrupt the participant, or attempt to sell or explain the product. Observers debate the participant’s actions in their presence. |
| Sensemaking | ||
| Notetaking | Studies are recorded or notes are taken for sharing with absent stakeholders. | Studies are not recorded or notes are not taken or available to the team. Test results are not documented. |
| Synthesis | Moderator debriefs with teammates immediately after each interview. Researcher looks for patterns across multiple participants and surfaces problems that affected several people. | Moderator reports the most memorable problems without conducting affinity mapping or some other analysis technique. |
| Presentation of findings | Researcher reports findings to team and stakeholders in an easy to follow, well prioritized way. They cite actual observations and voices where possible. | Researcher presents team a “basket of issues” or an unprioritized laundry list of potential changes. |
| Incorporation of findings | Product team translates findings into future user stories or product refinements. | |
A content heuristic is the specific, identifiable, and measurable qualities that content should have in order to achieve positive user experiences.
It is not enough to say that content should be ‘clear;’ instead it is more helpful to describe what clear content means for the user experience and how it should be demonstrated. Content heuristics are used to evaluate how well we are producing clear content and can be used during a content audit and user testing. With this in mind, here’s how we are defining clear, actionable, and accurate content.
| Indicator | Good sign | Bad sign |
|---|---|---|
| Clear content | ||
| Helpful, contextual titles | Use topic headings and subheadings to organize information and one main idea at a time.
Write specific navigation and page titles, headings, and labels that indicate what information users can expect to find and make it easier for them to pinpoint an answer |
Headings and sections have no hierarchy; navigation titles are general or repeated throughout the page (e.g. “read more”) |
| Concise, meaningful paragraphs | Page summary paragraph at the top of each web page that tells you exactly what information can be found on the page.
Simple paragraphs, no more than 3-5 sentences long, that describe the main idea and most important points of a process or task. Shorter sentences that communicate one idea or step at a time. |
No page summary or a summary that’s more than half as long as the rest of the page.
Complex paragraphs longer than 5 sentences long that digress or include information that does not support the main idea. Long sentences that include multiple sub-clauses or communicate too many steps or ideas at a time. |
| Consistent formatting and styling | Numbered or bulleted lists when describing step-by-step instructions.
More white space on the page and visually less text-heavy. Consistent standards for the application of text bolding, underlining, and italicizing. |
Instructions not broken down by step. More text than white space on a page.
Inconsistent use of bolding, underlining, or italicizing. |
| Use of plain language techniques | Break up dense and lengthy text by creating new topic pages.
Target a readability grade level and score under 12. Avoid using acronyms. When an acronym is necessary, define and spell it out the first time you use it. |
Pages that cover multiple topics.
Reading the page requires a readability grade level and score above 12. Content uses jargon or acronyms that are not spelled out. |
| Actionable content | ||
| Specific call-to-action | Direct users to a person or point of contact to get help if they need help completing a task.
Direct users to the tools, documents, or materials they need to complete a task. |
Users don’t have a clear path to help when they are likely to need it.
Access to tools, documents, or materials needed to complete a task are not included. |
| Instructive sentences | Provide information that tells users how to complete a task.
Indicate what the next steps are once a task is complete. Explain who is responsible for completing a task or managing a process, and what the user is required to do compared to other stakeholders involved in the process. |
Information about how to complete tasks is not provided.
Next steps after the task are not provided. Responsibility for completing a task or managing a process is not explained, and what the user is required to do compared to other stakeholders involved is not differentiated. |
| Illustrate and/or outline the timeline | Communicate about a task or process’ overall timeline, including deadlines, dependencies, duration, and any other time-sensitive activities.
Inform users where they are in the process. Inform users what done or successful task completion looks like. |
Description of tasks or processes include no or not enough information about timelines, deadlines, dependencies, duration, or other time-sensitivity.
No information about what successful task completion looks like is provided. |
| Accurate content | ||
| Consistency across channels | The information we provide is consistent with policies at all levels | Information is provided that conflicts with or is inconsistent with policies at all levels |
| Verifiable information | The correct point of contact information (e.g., name of person, phone, email, etc) is listed to answer questions and learn more. | Incorrect contact information is provided, or no contact information is provided. |
| Timely and trustworthy information | The user can trust and identify that the information is up-to-date. | The user cannot be sure that the information is up-to-date. |
The following indicators are intended to help stakeholders understand whether the product team has the leadership it needs in order to be successful.
“Product personnel” for the TDRS project include the OFA’s Product Owner and the vendor’s Facilitator. Between these two, responsibly for the items below may be shared. Ultimately, both should understand a problem space and articulate a vision for the product. The Product Owner should serve as a proxy for end users, provide day-to-day prioritization, and be the ultimate decision maker. The Facilitator on the other hand, should prioritize making sure the team is always focused on the right priorities and removing blockers for the team.
| Indicator | Good sign | Bad sign |
|---|---|---|
| People operations | ||
| Availability | Product personnel are allocated to work on the project (at least 50% time) for the duration of the engagement | Have other full time roles; are expected to treat the product role as “other duties as assigned” |
| Foundational knowledge and skills | ||
| Product basics | Is trained in core product management practices most valuable to them (e.g. roadmapping and strategic product thinking, scrum, cross-functional team management) | Participate only as a subject matter expert; do not lead the team |
| Demonstrate a bias toward action over philosophy, testing hypotheses, learning from failure, embrace experimentation, iteration, and uncertainty, etc. | Issues are frequently blocked because product can’t make a decision; they prioritize leadership consensus over experimentation and hypothesis testing | |
| Leadership | ||
| Feedback culture | Facilitate a culture where team members feel comfortable sharing feedback about all aspects of the project | Team members do not offer feedback; retros produce no useful insights |
| Cross-functional teams | Facilitate discussions around tradeoffs to design and engineering decisions, and seek team’s consensus on the best path forward | Team functions are siloed; how product decisions may impact dependencies among design, development, and architecture are not surfaced in time to inform decisions |
| Decision making | Make decisions about the product | Defer major product decisions to others on the team |
| Communication and collaboration | The team has a shared understanding of where, when, and how to communicate with each other; Key stakeholders know how they can contribute to the project as well as how to effectively give feedback to the project team | Team communication is not happening; people don’t know where to look or who to turn to with questions about the work or the product |
| Self care | ||
| Learning goals | Attend professional development trainings, conferences and other activities that support professional development goals; read professional journals and articles related to product practice and attends professional development trainings | Do not attend or engage in coaching sessions; are not aware of the current practices, developments, or debates in the field |
| Work-life balance | Set clear boundaries with their time; only works 40 hours week | |
| Delivery | ||
| Situational awareness | Product personnel understand and can explain: the state of the product, major implementation decisions made during product development, technologies used to build the product, work needed to further develop the product. | Are not aware of major implementation decisions, cannot articulate the product’s trajectory |
| Scoping | Can identify the Minimally Viable Product (MVP) from the roadmap with a focus on “must haves,” or non-negotiable functionality that will deliver a high value to end users | Are reactive to feature requests from stakeholders and users; scope an MVP that is overly complicated; MVP scope significantly expands, delaying release |
| Security- and privacy-minded | Use authorized tools/platforms (via FedRAMP, when possible) to support desired build methodology and get an ATO if necessary | Does not ask or assess whether ATOs are needed; uses and/or allows the use of unauthorized tools; is not aware of how ATO, SORN and/or other compliance processes may impact the product roadmap; |
| Measures success | Goals, outcomes, and targets have appropriate, realistic measures indicating how we’ll know when they’re met; Uses these metrics to forecast and adjust the trajectory of the project | Goals and metrics are not identified or adhered to; goals and metrics are not achievable or specific and/or are not directly tied to measures of user value |