Successful shared mobility projects require a clear vision of goals and associated performance metrics to measure their progress and outcomes.  One reason why goals and performance metrics are particularly important for shared mobility projects is that these projects often aim to address complex and systemic issues, such as equity, community engagement, accessibility, or sustainability, that require clear direction and planning to see progress. Another reason is the reality that shared mobility projects often work within a constrained budget- the goals and metrics should be deliberate and intentional, using the project’s finances efficiently. The process of setting goals, measuring performance metrics and evaluating the outcomes is an important step to staying organized and effective. 

This learning module first includes definitions for common terms related to setting goals and performance metrics, it then offers an organizational tool for the goal setting process along with important considerations for each step. Following this, the module uses five common goal areas for shared mobility projects – Accessibility, Usership, Community Engagement, Environmental Sustainability, and Connectivity – to provide example goals, metrics and evaluation methods specific to each. It hopes to serve as a mode-agnostic framework for shared mobility projects from goal setting through evaluation. 

• Data Elements indicate the specific information within the data type that the project requires such as ‘trip purpose’ or ‘monthly cost of bus operations’.
• Data Sources specifies the names of the sources or partners that either currently measure or plan to measure the data required such as the transit agency, private operator, or users involved with the project.
• Data Types indicate the general classification of data required such as survey data or economic data.
• Evaluation Hypothesis indicates what should happen if the project goals are met. 
• Performance Metrics indicate what pieces of information will be measured to demonstrate the impact of the project. 
• Project Goals or Objectives indicate what the project is trying to achieve. 

It is important to establish a way to organize the process of developing the project goals and performance metrics to evaluation and reporting. Shared mobility specifically, an innovative and emerging field that is continually evolving and changing, benefits from establishing an intended methodology from the onset. A logic model is an organizational framework that helps to visualize goals and connect the dots between goals, evaluation hypotheses, performance metrics, data types and data sources. A logic model will help identify any gaps in information and ensure that the methodology and flow of the project goals are viable. There are a variety of examples and structures of logic models; below is one example of an Evaluation Logic Model used in the MOD Sandbox Demonstrations Evaluations [5], specifically LA and Puget Sound’s First/Last Mile project partnership. For each Project Goal, the model includes an Evaluation Hypothesis, Performance Metrics, Data Types, and Data Sources. Other columns that can be added to the logic model to make it more comprehensive include identifying the Data Elements or the Period of Data Collection.

Evaluation Logic Model taken from Mobility on Demand (MOD) Sandbox Demonstrations Independent Evaluation: Los Angeles County and Puget Sound MOD First and Last Mile Partnership with Via

This section provides general considerations for each step in the process from setting goals, to creating and evaluating performance metrics using different data types and sources, to reporting back to key stakeholders on progress and outcomes. 

Project Goals: 

Defining the goals and objectives is a critical step in the early phases of a project. It is common for a project to have many goals depending on the scope of the work. To identify project goals, it can be helpful to think about what type of project it is. The type of project will inform the goals and objectives and the subsequent data required to measure it. Below are a couple of considerations for goal setting.

• Needs Assessment: A needs assessment is a process that includes a project’s key stakeholders and the community to identify what the needs of the community are to best determine the project’s direction and goals. This is a necessary step for new projects to understand what products and services will be useful to the community and can help shape the scope of work. To get a clear picture of the project area, it is helpful to create a geographic profile that not only provides demographic information of the community but also a baseline of its mobility, including variables such as accessibility, reliability or affordability. The Clean Mobility Options Needs Assessment Data Collection Guide offers specific data sources to create a mobility baseline based on accessibility, reliability and affordability can be found here. The results of the needs assessment should directly inform a project’s goals.                                                                                                                                       
• Goals setting criteria: Goals can vary in terms of their specificity. It may make sense for a project to have overarching, long term goals for a project, while also creating more detailed, interim or sub-objective goals to serve as checkpoints for the larger ones. SMART goals are a common goal setting criteria that stands for specific, measurable, attainable, relevant, and time-based [6].  

Performance Metrics, Data Types and Sources

Establishing performance metrics associated with the project goals involves thinking through the project’s desired outcomes and impact and identifying the appropriate data or information to measure those criteria. Following are some considerations to think through when developing performance metrics. 

• Data required: The data required to evaluate and measure a goal should be identified when establishing goals. Access to data should not necessarily deter from striving for that goal, however, it is helpful to set expectations early and understand if more innovative solutions to measure the outcomes are necessary. Helpful questions to think about when identifying the data required are: What do you need to learn? What types of analysis are required and why? What types of data would be needed? SUMC’s White Paper, Objective-Driven Data Sharing for Transit Agencies in Mobility Partnerships, can serve as a guide on the importance of defining the data needed early in a project to make sure that these are negotiated in the data sharing agreement between the agency and private operator.
• Qualitative or quantitative data: Both qualitative and quantitative data are important pieces of information to include in performance metrics. They complement each other and can both offer powerful insights. An example of qualitative data could be the information gathered from stakeholder interviews and a quantitative data example could be the number of transit trips taken.
• Common Data Types: Examples include survey data, ridership and activity data, payment data, fuel usage data, crime statistics data, or stakeholder interview data [5]. Survey data can come from ‘Before and After Surveys’ or ‘Recent Trip Surveys’.
• Common Data Sources: Key stakeholders such as a private operator, local transit agency, the payment company, the local MPO and/or Council of Governments, or surveys/on the ground observations are common date sources. 
• Data dependencies and interdependencies: This is the concept that there may be multiple performance metrics for one goal or conversely, one performance metric may meet more than one goal. Data dependencies and interdependencies help to prioritize what data is most important to collect. Data with higher counts of dependent metrics are the most important to collect as they inform the greatest number of metrics.
• Feasibility Analysis: A data feasibility analysis can help to identify how difficult the data will be to attain with ratings ranging from high to infeasible. ‘High’ feasibility indicates that the data are currently measured by a stakeholder/entity, ‘moderate’ feasibility means that the data are currently measured by external data, ‘low’ feasibility denotes that the data are currently not measurable but would be measurable in the future with insignificant to moderate additional effort and ‘infeasible’ means the data are currently not measurable and would be measurable in the future with significant additional effort that would require policy and/or regulatory actions [3]. Questions used in a feasibility analysis include [3]:
  • Does the data exist?
  • Are the data available?
  • What is the feasibility of obtaining the data?
  • What are the constraints and the level of those constraints associated with available data?
  • What would be the format and unit of a given data element?
  • What are the data sets that are currently unavailable, but in development phases by the agencies or stakeholders?
  • What are the data that are not available?
• Data Collection Period: This involves the timing and frequency of the data required. Common project data needs include those described below. 
  • Transportation Trends: Archived, Post-Processed Data 
  • Service Planning: Archived, Post-Processed Data 
  • Service Delivery, Effectiveness, Accountability: Trip Data involving, operations data (Pickup and drop-off times, wait times, travel times, etc.), accounting data, and auditing data
  • Integrated Trip-Planning and Payment: Real-Time Data
• Data Standards: As the technical capability of public agencies and mobility providers continues to improve, data specifications are becoming more popular as a way to more easily accelerate mobility partnerships across many types of agencies and cities. While specifications such as GTFS, GBFS, and TDS have transformed how transit and micromobility services communicate on the back end to more easily enable trip planning/Mobility-as-a-Service tools for travelers, they do not directly address how data about trips or rides should be collected to measure the success of mobility projects. One way to collect data about micromobility trips in particular is the Mobility Data Specification (MDS). However, MDS is intended more as a regulatory tool and may not be appropriate for your project, especially since the specification requires very granular data collection that may not be suitable for your specific project or your agency’s technical capacity. More generally, SAE has developed standards defining automation levels and taxonomy for micromobility technology and shared mobility services that help agencies more clearly communicate the intent or scope of their service.
• Establishing data sharing agreements: Despite advances in data availability the private operator may be reluctant to share for proprietary and confidentiality reasons. If the project goal requires specific data, it is important to explain their value and how they will be used to meet a project’s goal. Data sharing agreements are helpful in these situations to articulate what information is needed and its intended use. As mentioned above, best practices for transit agencies can be found in SUMC’s White Paper, Objective-Driven Data Sharing for Transit Agencies in Mobility Partnerships.
• Data Risk Management: This is associated with the data collection process. For example when using survey data as the survey type for a performance metric, there is a risk that low survey response rate does not lead to statistically significant results [5]. Some of the main risks include maintaining the data schedule, performing data quality assurance spot checks, data sufficiency, and discussed below, data anonymity.
• Data Protection Plan: Whether the source is a partner or a third party, the data must be handled and stored appropriately. The primary concern on all sides is data management and security. This could affect collection, formatting, storage, archiving, access/authorization protocols, cybersecurity, reporting (internal and external), and compatibility with existing systems. Data handling and security protocols must be discussed thoroughly before forming partnerships and clearly defined in agreements, while adhering to the regulations regarding data and privacy at the local, state, and federal levels. 

Evaluation

There are a variety of ways to evaluate the progress and outcomes of a project, see examples listed below: 

• Types of analysis: Survey analysis, a time series analysis of ridership data or access data, a cross-sectional analysis of unlinked trip data and other activity data, activity data analysis before and after program implementation, a spatial analysis of riders and activity data before and after the program implementation, revenue analysis, or a summary of expert interviews [5]. 

Reporting

How and when a project disseminates its results and findings to its constituents can be a critical component to retaining funding and showing the value of the implementation. 

• Frequency of Reporting: Analyzing and reporting should be an ongoing procedure to keep key stakeholders engaged and aware of progress and ensure the initial results are tracking to anticipated goals. The frequency can vary from project to project but it is common to report back to a funder on a monthly or quarterly basis. With this in mind, it is important to consider a succinct format to track and report out on key elements. For example, projects focused on community engagement may create a survey to give to their steering committee on a monthly basis with a few of the questions and answers pulled into a monthly report. Equally important for any pilot projects is the final report that should include an executive summary and details of the demonstration, evaluation hypotheses, data collected, analysis performed, findings, and results. The report can also utilize a mix of exhibits including tables, graphs, and charts.
• Distribution of Reporting: Common recipients of shared mobility project reports include the funder of the pilot project, such as the Federal Transit Administration, the Board members of the organization if applicable, key partners, and the community. Reporting out to the community is an important component of all projects but specifically for those that have held community engagement activities during the planning and implementation phases. This can help build trust between the community and the project provider, showing the community the results and impact of their efforts.
• Example evaluation reports:

• • Chicago 2019 Scootershare Evaluation Report 
  • TriMet OpenTripPlanner MOD Sandbox Quarterly Report 
  • CTAA’s Inclusive Planning Guide to Evaluation

The next section provides goal and performance metric information specific to five goal areas that are commonly found across shared mobility projects: Accessibility, Usage, Environmental, Connectivity, and Equity/Community Engagement. Projects may be focused on any combination of these and more. Each goal area will extrapolate on a specific goal and demonstrate how it might be used in a logic model. To root this exercise in a real world example, we used a hypothetical shared mobility project:

Hypothetical Project: A microtransit shared mobility pilot project, called Mobility 365, located in the suburbs of a large city in the United States is focused on improving the first/last mile connections at the end of a light rail system. The shuttle will be wheelchair accessible, and hopes to reduce single occupancy vehicle commutes, increase public transit utilization, and increase connectivity. The project is led by the local transit agency, leveraging the support, knowledge and expertise of two community based organizations and will procure the vehicle from a private operator. Capital funding for this project is supplied primarily through a federal grant reserved for projects that have an emphasis on equity and sustainability. For the purposes of this exercise, we will call the transit agency, X Area Rapid Transit (XART), and the private operator, Microtransit Inc.  

Accessibility 

Shared mobility projects must meet ADA equivalent service requirements.  However, projects focused on accessibility as a goal are looking for transit oriented solutions to improve access to essential human services for all people but specifically those with disabilities. Example accessibility goals could include improving mobility for persons with disabilities, complying with ADA equivalent level of service requirements, improving mobility for users of wheelchairs, and enhancing trip planning methods for persons with disabilities. Common performance metrics associated with these accessibility goals could include average wait time/planning time/travel time/travel distance, Wheelchair Accessible Vehicle (WAV) trip requests, and total WAV trips provided. 

Below is an example logic model for the specific goal to ‘Improve mobility for persons with disabilities’ within the context of the microtransit project example. 

For an accessible version, please click here.

Here are a few examples that have looked at accessibility:

1. Kansas City RideKC Freedom On-Demand program
2. MBTA The RIDE 
3. WMATA Mobilities Ride

Usage

Usership focuses on increased utilization of a particular service or product and is often achieved through a combination of increased operational efficiency and improved user experience. Common goals include reducing travel times, reducing wait times, increasing public transit use, increasing transit reliability, or improving rider satisfaction. Common performance metrics include measuring travel times and wait times, count of unlinked trips at selected stations, count of unique users, riders per vehicle service hour, change in reported transit ridership due to XART implementation, 

Below is a logic model for the specific goal to ‘Increase public transit utilization’ within the context of the microtransit project example. 

For an accessible version, please click here.

Below are a few project examples where ridership was considered: 

1. Norwalk | Wheels2U Microtransit Service
2. Arlington, TX Microtransit Pilot
3. SW Prime Microtransit Service

SW Prime Microtransit, Source: SouthWest Transit

Environmental

Goals related to the environment use a variety of tactics, however, they all revolve around improving the quality of the environment through innovative shared mobility projects. Common goals and objectives for environmental goals are to improve air quality, reduce congestion, reduce greenhouse gas emissions, reduce single occupancy vehicle (SOV) VMT, improve electric vehicle usage, plan and build pedestrian and micromobility infrastructure, increase utilization of parking spaces by carpool vehicle, or increase public transit use. Common performance metrics might involve measuring GHG emissions, measuring travel behavior such as the number of single-occupancy vehicles, measuring the number of carpooling riders, or measuring the number of verified carpool vehicles at a train station. 

Below is a logic model for the specific goal to ‘Reduce vehicle miles travelled and greenhouse gas emissions from travel’ within the context of the microtransit project example. 

For an accessible version, please click here.

Here are a few examples of projects that placed value on the environmental component of their project:

1. Seattle Via to Transit Pilot 
2. BlueLA Carsharing Pilot 
3. BART Integrated Carpool to Transit Access Program

Connectivity

Connectivity goals relate to improving the connections using a multi-modal transit network solution. Common goals involve improving user perceived connectivity throughout the transit network, improving first/last mile connections, increasing public transit utilization, and improving transit connections with neighboring communities. Performance metrics associated with these goals include measuring the number of transit connections with neighboring communities, measuring the number of first/last mile trips, or measuring the number of public transit riders after implementing the project.

Below is a logic model for the specific goal to ‘Improve first/last mile connectivity’ within the context of the microtransit project example. 

For an accessible version, please click here.

Here are a few examples of projects that looked at connectivity:

1. Lone Tree Link 
2. SW Prime
3. DART Microtransit Pilot Project

Community Engagement

Community engagement goals often involve establishing a community engagement process using inclusion planning processes that reflects the diversity of the community, reaching the ‘collaboration’ stage on the spectrum of participatory planning, and creating a sustainable community engagement framework to be used for future projects. In general, performance metrics for community engagement goals may have more qualitative data than other goals and often revolve around measuring levels of equity and participant satisfaction with the process. To identify what performance metrics to use, it is helpful to think about what ‘success’ looks like for this goal, which is often something observable or measurable. Additionally, since community engagement is a process, performance indicators should evaluate both the process in terms of the methods and tools used and the results, as in the outcomes of the process [6]. 

Due to the fact that there are a number of community engagement strategies based on project goals and resources, the evaluative data collection may include a combination of face-to-face, written, and online feedback. Equally, there are a variety of evaluative methods such as a human centered design technique called journey mapping, or another common analysis called an ‘equity analysis’. An ‘equity analysis’ is an evaluation of the demographics reached during a project phase [6]. It compares the demographics of a survey or meeting to the demographics of the project area established in the goal setting stage. This comparison can help to identify which groups are underrepresented to help the team develop targeted outreach for future activities. This analysis is most effective when done after each phase of the project rather than every activity as the results are less significant for a single meeting. This analysis can be compiled into a single document at the end of a project so the team can evaluate what was effective or not to help direct future projects. Questions to ask during an equity analysis is [6]: 

• Which demographics groups participated the most? 
• Which demographic groups were underrepresented? 
• What outreach activities were conducted during this activity or phase? 
• What communications were distributed during this activity or phase? 
• What outreach and communication methods can I use to reach the underrepresented populations in the next activity or phase of the project? 

Below is a logic model for the specific goal to ‘Improve inclusive planning through community engagement efforts’ within the context of our microtransit project example. 

For an accessible version, please click here.

Here are a few project examples that have focused on equity and community engagement:

1. Chicago E-Scooter Pilot 
2. BlueLA Carsharing Pilot 
3. Community Engagement Case Studies

There are several challenges and considerations to navigate when establishing project goals and identifying the data required to support them. A few include Data Sharing, Data Privacy, Data Competitiveness, Data Aggregation, and Roles and Responsibilities. For more information on these challenges, read our overview titled Challenges for Establishing Goals and Metrics. 

This Learning Module strives to provide key considerations about the process of setting goals all the way through reporting on results. It uses these considerations for a hypothetical microtransit pilot program and five common goal areas that are often considered for shared-use and mobility on demand projects. The paper ends with identifying a few challenges that are likely to arise in this process to help set expectations and offer ways to mitigate them. With many new and innovative shared mobility projects underway, it is critical to establish a measurable, data-driven approach to understand what worked well and what can be improved upon for future projects. 

1. SUMC Microtransit Learning Module
2. Clean Mobility Options Needs Assessment Data Collection Guide 
3. FTA Mobility Performance Metrics for Integrated Mobility and Beyond Report 
4. TCRP Synthesis 141: Microtransit or General Public DRT Services 
5. MOD Sandbox Demonstration Evaluation Reports 
6. Community Engagement Process Development Public Participation Playbook 
7. Using Human Centered Design to Advance Civic Engagement in Nonprofits
8. Objective Data Driven data Sharing for Transit Agencies in Mobility Partnerships 
9. De Montjoye, Y. A., Hidalgo, C. A., Verleysen, M., & Blondel, V. D. (2013). Unique in the crowd: The privacy bounds of human mobility. Scientific reports, 3, 1376