Successful shared mobility projects require a clear vision of goals and associated performance metrics to measure their progress and outcomes. One reason why goals and performance metrics are particularly important for shared mobility projects is that these projects often aim to address complex and systemic issues, such as equity, community engagement, accessibility, or sustainability, that require clear direction and planning to see progress. Another reason is the reality that shared mobility projects often work within a constrained budget- the goals and metrics should be deliberate and intentional, using the project’s finances efficiently. The process of setting goals, measuring performance metrics and evaluating the outcomes is an important step to staying organized and effective.
This learning module first includes definitions for common terms related to setting goals and performance metrics, it then offers an organizational tool for the goal setting process along with important considerations for each step. Following this, the module uses five common goal areas for shared mobility projects – Accessibility, Usership, Community Engagement, Environmental Sustainability, and Connectivity – to provide example goals, metrics and evaluation methods specific to each. It hopes to serve as a mode-agnostic framework for shared mobility projects from goal setting through evaluation.
It is important to establish a way to organize the process of developing the project goals and performance metrics to evaluation and reporting. Shared mobility specifically, an innovative and emerging field that is continually evolving and changing, benefits from establishing an intended methodology from the onset. A logic model is an organizational framework that helps to visualize goals and connect the dots between goals, evaluation hypotheses, performance metrics, data types and data sources. A logic model will help identify any gaps in information and ensure that the methodology and flow of the project goals are viable. There are a variety of examples and structures of logic models; below is one example of an Evaluation Logic Model used in the MOD Sandbox Demonstrations Evaluations [5], specifically LA and Puget Sound’s First/Last Mile project partnership. For each Project Goal, the model includes an Evaluation Hypothesis, Performance Metrics, Data Types, and Data Sources. Other columns that can be added to the logic model to make it more comprehensive include identifying the Data Elements or the Period of Data Collection.
This section provides general considerations for each step in the process from setting goals, to creating and evaluating performance metrics using different data types and sources, to reporting back to key stakeholders on progress and outcomes.
Project Goals:
Defining the goals and objectives is a critical step in the early phases of a project. It is common for a project to have many goals depending on the scope of the work. To identify project goals, it can be helpful to think about what type of project it is. The type of project will inform the goals and objectives and the subsequent data required to measure it. Below are a couple of considerations for goal setting.
Performance Metrics, Data Types and Sources
Establishing performance metrics associated with the project goals involves thinking through the project’s desired outcomes and impact and identifying the appropriate data or information to measure those criteria. Following are some considerations to think through when developing performance metrics.
Evaluation
There are a variety of ways to evaluate the progress and outcomes of a project, see examples listed below:
Reporting
How and when a project disseminates its results and findings to its constituents can be a critical component to retaining funding and showing the value of the implementation.
The next section provides goal and performance metric information specific to five goal areas that are commonly found across shared mobility projects: Accessibility, Usage, Environmental, Connectivity, and Equity/Community Engagement. Projects may be focused on any combination of these and more. Each goal area will extrapolate on a specific goal and demonstrate how it might be used in a logic model. To root this exercise in a real world example, we used a hypothetical shared mobility project:
Hypothetical Project: A microtransit shared mobility pilot project, called Mobility 365, located in the suburbs of a large city in the United States is focused on improving the first/last mile connections at the end of a light rail system. The shuttle will be wheelchair accessible, and hopes to reduce single occupancy vehicle commutes, increase public transit utilization, and increase connectivity. The project is led by the local transit agency, leveraging the support, knowledge and expertise of two community based organizations and will procure the vehicle from a private operator. Capital funding for this project is supplied primarily through a federal grant reserved for projects that have an emphasis on equity and sustainability. For the purposes of this exercise, we will call the transit agency, X Area Rapid Transit (XART), and the private operator, Microtransit Inc.
Accessibility
Shared mobility projects must meet ADA equivalent service requirements. However, projects focused on accessibility as a goal are looking for transit oriented solutions to improve access to essential human services for all people but specifically those with disabilities. Example accessibility goals could include improving mobility for persons with disabilities, complying with ADA equivalent level of service requirements, improving mobility for users of wheelchairs, and enhancing trip planning methods for persons with disabilities. Common performance metrics associated with these accessibility goals could include average wait time/planning time/travel time/travel distance, Wheelchair Accessible Vehicle (WAV) trip requests, and total WAV trips provided.
Below is an example logic model for the specific goal to ‘Improve mobility for persons with disabilities’ within the context of the microtransit project example.
For an accessible version, please click here.
Here are a few examples that have looked at accessibility:
Usage
Usership focuses on increased utilization of a particular service or product and is often achieved through a combination of increased operational efficiency and improved user experience. Common goals include reducing travel times, reducing wait times, increasing public transit use, increasing transit reliability, or improving rider satisfaction. Common performance metrics include measuring travel times and wait times, count of unlinked trips at selected stations, count of unique users, riders per vehicle service hour, change in reported transit ridership due to XART implementation,
Below is a logic model for the specific goal to ‘Increase public transit utilization’ within the context of the microtransit project example.
For an accessible version, please click here.
Below are a few project examples where ridership was considered:
SW Prime Microtransit, Source: SouthWest Transit
Environmental
Goals related to the environment use a variety of tactics, however, they all revolve around improving the quality of the environment through innovative shared mobility projects. Common goals and objectives for environmental goals are to improve air quality, reduce congestion, reduce greenhouse gas emissions, reduce single occupancy vehicle (SOV) VMT, improve electric vehicle usage, plan and build pedestrian and micromobility infrastructure, increase utilization of parking spaces by carpool vehicle, or increase public transit use. Common performance metrics might involve measuring GHG emissions, measuring travel behavior such as the number of single-occupancy vehicles, measuring the number of carpooling riders, or measuring the number of verified carpool vehicles at a train station.
Below is a logic model for the specific goal to ‘Reduce vehicle miles travelled and greenhouse gas emissions from travel’ within the context of the microtransit project example.
For an accessible version, please click here.
Here are a few examples of projects that placed value on the environmental component of their project:
Connectivity
Connectivity goals relate to improving the connections using a multi-modal transit network solution. Common goals involve improving user perceived connectivity throughout the transit network, improving first/last mile connections, increasing public transit utilization, and improving transit connections with neighboring communities. Performance metrics associated with these goals include measuring the number of transit connections with neighboring communities, measuring the number of first/last mile trips, or measuring the number of public transit riders after implementing the project.
Below is a logic model for the specific goal to ‘Improve first/last mile connectivity’ within the context of the microtransit project example.
For an accessible version, please click here.
Here are a few examples of projects that looked at connectivity:
Community Engagement
Community engagement goals often involve establishing a community engagement process using inclusion planning processes that reflects the diversity of the community, reaching the ‘collaboration’ stage on the spectrum of participatory planning, and creating a sustainable community engagement framework to be used for future projects. In general, performance metrics for community engagement goals may have more qualitative data than other goals and often revolve around measuring levels of equity and participant satisfaction with the process. To identify what performance metrics to use, it is helpful to think about what ‘success’ looks like for this goal, which is often something observable or measurable. Additionally, since community engagement is a process, performance indicators should evaluate both the process in terms of the methods and tools used and the results, as in the outcomes of the process [6].
Due to the fact that there are a number of community engagement strategies based on project goals and resources, the evaluative data collection may include a combination of face-to-face, written, and online feedback. Equally, there are a variety of evaluative methods such as a human centered design technique called journey mapping, or another common analysis called an ‘equity analysis’. An ‘equity analysis’ is an evaluation of the demographics reached during a project phase [6]. It compares the demographics of a survey or meeting to the demographics of the project area established in the goal setting stage. This comparison can help to identify which groups are underrepresented to help the team develop targeted outreach for future activities. This analysis is most effective when done after each phase of the project rather than every activity as the results are less significant for a single meeting. This analysis can be compiled into a single document at the end of a project so the team can evaluate what was effective or not to help direct future projects. Questions to ask during an equity analysis is [6]:
Below is a logic model for the specific goal to ‘Improve inclusive planning through community engagement efforts’ within the context of our microtransit project example.
For an accessible version, please click here.
Here are a few project examples that have focused on equity and community engagement:
There are several challenges and considerations to navigate when establishing project goals and identifying the data required to support them. A few include Data Sharing, Data Privacy, Data Competitiveness, Data Aggregation, and Roles and Responsibilities. For more information on these challenges, read our overview titled Challenges for Establishing Goals and Metrics.
This Learning Module strives to provide key considerations about the process of setting goals all the way through reporting on results. It uses these considerations for a hypothetical microtransit pilot program and five common goal areas that are often considered for shared-use and mobility on demand projects. The paper ends with identifying a few challenges that are likely to arise in this process to help set expectations and offer ways to mitigate them. With many new and innovative shared mobility projects underway, it is critical to establish a measurable, data-driven approach to understand what worked well and what can be improved upon for future projects.