Definitions

• Data Elements: Indicate the specific information within the data type that the project requires, such as trip purpose or monthly cost of bus operations.
• Data Sources: Specify the names of the sources or partners that currently measure or plan to measure the required data, such as the transit agency, private operator, or users involved with the project.
• Data Types: Indicate the general classification of data required, such as survey or economic data.
• Evaluation Hypothesis: Indicates the next steps after the project achieves its goals.
• Performance Metrics: Indicate what information will be measured to demonstrate the project’s impact.
• Project Goals or Objectives: Indicate what the project is trying to achieve.

Logic Models

It is essential to establish a way to develop the project goals and performance metrics for evaluation and reporting. Shared mobility, specifically benefits from establishing an intended methodology from the onset. A logic model is an organizational framework that helps visualize and find the association between goals, evaluation hypotheses, performance metrics, data types, and data sources. A logic model will help identify gaps in information and ensure that the methodology and flow of the project goals are viable. There are a variety of examples and structures of logic models; below is one example of an Evaluation Logic Model used in the MOD Sandbox Demonstrations Evaluations [5], specifically LA and Puget Sound’s First/Last Mile project partnership. For each Project Goal, the model includes an Evaluation Hypothesis, Performance Metrics, Data Types, and Data Sources. Other columns that can be added to the logic model to make it more comprehensive include identifying the Data Elements or the Period of Data Collection.

Evaluation Logic Model taken from Mobility on Demand (MOD) Sandbox Demonstrations Independent Evaluation: Los Angeles County and Puget Sound MOD First and Last Mile Partnership with Via

Project Goals:

Defining the goals and objectives is a critical step in the early phases of a project. It is common for a project to have many goals depending on the scope of the work. To identify the project goals, we first need to think about what type of project it is. The type of project will inform the goals and objectives and the subsequent data required to measure it. Below are a couple of considerations for goal setting.

Needs Assessment: A needs assessment is a process that includes a projects’ key stakeholders and the community to identify what the needs of the community are to best determine the project’s direction and goals. This step is necessary for new projects to understand what products and services will be useful to the community and help shape the work scope. To get a clear picture of the project area, we need to set a geographic profile that provides demographic information of the community and a baseline of its mobility, including variables such as accessibility, reliability, or affordability. The Clean Mobility Options Needs Assessment Data Collection Guide offers specific data sources to create accessibility, reliability, and affordability-based mobility baseline. The results of the needs assessment should directly inform a projects’ goals.

Goals setting criteria: Goals can vary in terms of their specificity. It may make sense for a project to have overarching, long-term goals while also creating more detailed, interim, or sub-objective goals to serve as checkpoints for the larger ones. SMART goals are common goal-setting criteria that are specific, measurable, attainable, relevant, and time-based [6]. 

Next Subsection

Performance Metrics, Data Type

Establishing performance metrics associated with the project goals involves thinking through the projects’ desired outcomes and impact and identifying the appropriate data or information to measure those criteria. Following are some considerations to think through when developing performance metrics.

Data required: The data required to evaluate and measure a goal should be identified when establishing goals. Access to data should not necessarily deter from striving for that goal. However, it is helpful to set expectations early and understand if more innovative solutions to measure the outcomes are necessary. Valuable questions to think about when identifying the data required are: What do you need to learn? What types of analysis are required and why? What types of data would be needed? SUMC’s White Paper, Objective-Driven Data Sharing for Transit Agencies in Mobility Partnerships, can guide the importance of defining the data required early in a project to ensure that these are negotiated in the data-sharing agreement between the agency and private operator.

Qualitative or quantitative data: Qualitative and quantitative data are important information to include in performance metrics. They complement each other and can both offer powerful insights. An example of qualitative data could be the information gathered from stakeholder interviews, and a quantitative data example could be the number of transit trips taken.

Common Data Types: Examples include survey data, ridership and activity data, payment data, fuel usage data, crime statistics data, or stakeholder interview data [5]. Survey data can come from ‘Before and After Surveys’ or ‘Recent Trip Surveys.’

Common Data Sources: Key stakeholders such as a private operator, local transit agency, the payment company, the local metropolitan planning organization (MPO) and/or Council of Governments, or surveys/on the ground observations are common data sources.

Data dependencies and interdependencies: This is the concept that there may be multiple performance metrics for one goal, or conversely, one performance metric may meet more than one goal. Data dependencies and interdependencies help prioritize what data is most important to collect. Data with higher counts of dependent metrics are the most important to collect as they inform the most significant metrics.

Feasibility Analysis: A data feasibility analysis can help identify how complex the data will attain with ratings ranging from high to infeasible.

• High feasibility. Indicates that the data are currently measured by a stakeholder/entity
• Moderate feasibility. It means that external data currently measure the data 
• Low feasibility. It denotes that the data are currently not measurable but would be measurable in the future with insignificant to moderate additional effort
• Infeasible. This indicates that the data is currently not measurable and would be measurable in the future with the significant extra effort that would require policy and/or regulatory actions [3]. 

Questions used in a feasibility analysis include [3]:

• Does the data exist?
• Are the data available?
• What is the feasibility of obtaining the data?
• What are the constraints and the level of those constraints associated with available data?
• What would be the format and unit of a given data element?
• What data sets are currently unavailable but in development phases by the agencies or stakeholders?
• What are the data that are not available?

Data Collection Period: This involves the timing and frequency of the data required. Common project data needs include those described below. 

• Transportation Trends: Archived, Post-Processed Data 
• Service Planning: Archived, Post-Processed Data 
• Service Delivery, Effectiveness, Accountability: Trip Data involving, operations data (Pickup and drop-off times, wait times, travel times, etc.), accounting data, and auditing data
• Integrated Trip-Planning and Payment: Real-Time Data

Data Standards: As the technical capability of public agencies and mobility providers continues to improve, data specifications are becoming more popular as a way to more easily accelerate mobility partnerships across many types of agencies and cities. While specifications such as GTFS, GBFS, and TDS have transformed how transit and micromobility services communicate on the back end to more easily enable trip planning/MaaS (Mobility as a Service) tools for travelers, they do not directly address how data about trips or rides should be collected to measure the success of mobility projects. One way to collect data about micromobility trips, in particular, is the Mobility Data Specification (MDS). However, MDS is intended more as a regulatory tool and may not be appropriate for your project, especially since the specification requires very granular data collection that may not suit your specific project or your agency’s technical capacity. More generally, SAE has developed standards defining automation levels and taxonomy for micromobility technology and shared mobility services that help agencies communicate the intent or scope of their service more clearly.

Establishing data sharing agreements: Despite advances in data availability, the private operator may be reluctant to share for proprietary and confidentiality reasons. If the project goal requires specific data, it is crucial to explain their value and how they will be used to meet a project’s goal. Data sharing agreements are helpful in these situations to articulate what information is needed and its intended use. As mentioned above, best practices for transit agencies can be found in SUMC’s White Paper, Objective-Driven Data Sharing for Transit Agencies in Mobility Partnerships.

Data Risk Management: This is associated with the data collection process. For example, when using survey data as the survey type for a performance metric, there is a risk that a low survey response rate does not lead to statistically significant results [5]. Some of the main risks include maintaining the data schedule, performing data quality assurance spot checks, data sufficiency, and discussed below, data anonymity.

Data Protection Plan: Whether the source is a partner or a third party, the data must be handled and stored appropriately. The primary concern on all sides is data management and security. This could affect collection, formatting, storage, archiving, access/authorization protocols, cybersecurity, reporting (internal and external), and compatibility with existing systems. Data handling and security protocols must be discussed thoroughly before forming partnerships and clearly defined in agreements while adhering to the regulations regarding data and privacy at the local, state, and federal levels.

Previous Subsection Next Subsection

Evaluation

There are various ways to evaluate the progress and outcomes of a project. See examples listed below: 

Types of analysis: Survey analysis, a time series analysis of ridership data or access data, a cross-sectional analysis of unlinked trip data and other activity data, activity data analysis before and after program implementation, a spatial analysis of riders and activity data before and after the program implementation, revenue analysis, or a summary of expert interviews [5].

Previous Subsection Next Subsection

Reporting

How and when a project disseminates its results and findings to its constituents can be a critical component to retaining funding and showing the value of the implementation.

Frequency of Reporting: Analyzing and reporting should be an ongoing procedure to keep key stakeholders engaged and aware of progress and ensure the initial results are tracking to anticipated goals. The frequency can vary from project to project, but it is common to report back to a funder monthly or quarterly. With this in mind, it is essential to consider a concise format to track and report out on key elements. For example, projects focused on community engagement may create a survey to give to their steering committee every month, including a few of the questions and answers pulled into a monthly report. The final report is equally important for any pilot project, including an executive summary and details of the demonstration, evaluation hypotheses, data collected, analysis performed, findings, and results. The report can also utilize a mix of exhibits, including tables, graphs, and charts.

Distribution of Reporting: Common recipients of shared mobility project reports include the funder of the pilot project, such as the Federal Transit Administration, the Board members of the organization, if applicable, key partners, and the community. Reporting to the community is an important component of all projects but specifically for those who have held community engagement activities during the planning and implementation phases. This process can help build trust between the community and the project provider, showing the community the results and impact of their efforts.

• Example evaluation reports:

• Chicago 2020 Scootershare Evaluation Report 
• TriMet OpenTripPlanner MOD Sandbox Quarterly Report 
• Santa Monica Shared Mobility Pilot Program 2019 Evaluation Report

Previous Subsection

Project Goals and Performance

Common Goal Areas

The next section provides goal and performance metric information specific to five goal areas that are commonly found across shared mobility projects: Accessibility, Usage, Environmental, Connectivity, and Equity/Community Engagement. Projects may be focused on any combination of these and more. Each goal area will extrapolate on a specific goal and demonstrate how it might be used in a logic model. We used a hypothetical shared mobility project to root this exercise in a real-world example.

Hypothetical Project: A microtransit shared mobility pilot project, called Mobility 365, located in the suburbs of a large city in the United States, focuses on improving the first/last mile connections at the end of a light rail system. The shuttle will be wheelchair accessible and hopes to reduce single-occupancy vehicle commutes, increase public transit utilization, and increase connectivity. The project is led by the local transit agency, leveraging two community-based organizations’ support, knowledge, and expertise, and will procure the vehicle from a private operator. Capital funding for this project is supplied primarily through a federal grant reserved for projects that emphasize equity and sustainability. For this exercise, we will call the transit agency, X Area Rapid Transit (XART), and the private operator Microtransit Inc.

Next Subsection

Accessibility

Shared mobility projects must meet ADA equivalent service requirements.  However, projects focused on accessibility as a goal are looking for transit-oriented solutions to improve access to essential human services for all people, specifically those with disabilities. Example accessibility goals could include improving mobility for persons with disabilities, complying with ADA equivalent level of service requirements, improving mobility for users of wheelchairs, and enhancing trip planning methods for persons with disabilities. Common performance metrics associated with these accessibility goals could include average wait time/planning time/travel time/travel distance, Wheelchair Accessible Vehicle (WAV) trip requests, and total WAV trips provided. 

Below is an example logic model for the specific goal to ‘Improve mobility for persons with disabilities’ within the context of the microtransit project example.

For an accessible version, please click here.

Here are a few examples that have looked at accessibility:

• Pace Partners with Uber to Offer Rides to Paratransit Customers
• DART Microtransit Pilot Program
• WMATA Mobilities Ride

Previous Subsection Next Subsection

Usage

Users focus on increased utilization of a particular service or product and are often achieved through a combination of increased operational efficiency and improved user experience. Common goals include reducing travel times, reducing wait times, increasing public transit use, increasing transit reliability, or improving rider satisfaction. Common performance metrics include measuring travel times and wait times, count of unlinked trips at selected stations, count of unique users, riders per vehicle service hour, change in reported transit ridership due to XART implementation, 

Below is a logic model for the specific goal to ‘Increase public transit utilization’ within the context of the microtransit project example.

Below are a few project examples where ridership was considered: 

• SW Prime Microtransit Service
• Norwalk | Wheels2U Microtransit Service
• Arlington, TX On-Demand Autonomous Shuttles

Previous Subsection Next Subsection

Environmental

Goals related to the environment use a variety of tactics. However, they all revolve around improving the quality of the environment through innovative shared mobility projects. Common environmental goals are to improve air quality, reduce congestion, greenhouse gas emissions, single-occupancy vehicle (SOV) vehicle miles traveled (VMT), improve electric vehicle usage, plan and build pedestrian and micromobility infrastructure, increase utilization of parking spaces by carpool vehicle, or increase public transit use. Common performance metrics might involve measuring GHG emissions, measuring travel behavior such as the number of SOVs, measuring the number of carpooling riders, or measuring the number of verified carpool vehicles at a train station. 

Below is a logic model for the specific goal to ‘Reduce vehicle miles traveled and greenhouse gas emissions from travel’ within the microtransit project example.

Here are a few examples of projects that placed value on the environmental component of their project:

• Seattle Via to Transit Pilot 
• BlueLA Carsharing Pilot 
• BART Integrated Carpool to Transit Access Program

Previous Subsection Next Subsection

Connectivity

Connectivity goals relate to improving the connections using a multi-modal transit network solution. Common goals involve improving user-perceived connectivity throughout the transit network, improving first/last mile connections, increasing public transit utilization, and improving transit connections with neighboring communities. Performance metrics associated with these goals include measuring the number of transit connections with neighboring communities, the number of first/last mile trips, or the number of public transit riders after implementing the project.

Below is a logic model for the specific goal to ‘Improve first/last mile connectivity within the context of the microtransit project example.

Here are a few examples of projects that looked at connectivity:

• Flex Ride Milwaukee
• SW Prime
• DART Microtransit Pilot Project

Previous Subsection Next Subsection

Equity/Community Engagement

Community engagement goals often involve establishing an inclusive planning process that reflects the community’s diversity, reaching the ‘collaboration’ stage on the spectrum of participatory planning, and creating a sustainable engagement framework to be used for future projects. In general, performance metrics for community engagement goals may have more qualitative data than other goals and often revolve around measuring levels of equity and participant satisfaction with the process. To identify the ideal performance metrics, we should think about what ‘success’ looks like for this goal, which is often observable or measurable. Additionally, since community engagement is a process, performance indicators should evaluate both the process in terms of the methods and tools used and the results, as in the outcomes of the process [6]. 

Since there are several community engagement strategies based on project goals and resources, the evaluative data collection may include a combination of face-to-face, written, and online feedback. Equally, there are a variety of evaluative methods, such as a human-centered design technique called journey mapping, or another common analysis called an ‘equity analysis.’ An ‘equity analysis’ evaluates the demographics reached during a project phase [6]. It compares the demographics of a survey or meeting to the demographics of the project area established in the goal-setting stage. This comparison can help to identify which groups are underrepresented to help the team develop targeted outreach for future activities. This analysis is most effective when done after each phase of the project rather than every activity as the results are less significant for a single meeting. This analysis can be compiled into a single document at the end of a project so the team can evaluate what was effective or not to help direct future projects. Questions to ask during an equity analysis is [6]: 

• Which demographic groups participated the most? 
• Which demographic groups were underrepresented? 
• What outreach activities were conducted during this activity or phase? 
• What communications were distributed during this activity or phase? 
• What outreach and communication methods can I use to reach the underrepresented populations in the next activity or phase of the project? 

Below is a logic model for the specific goal to ‘Improve inclusive planning through community engagement efforts’ within the context of our microtransit project example.

Here are a few project examples that have focused on equity and community engagement:

• Universal Basic Mobility – Oakland
• Chicago E-Scooter Pilot 
• BlueLA Carsharing Pilot 
• Community Engagement Case Studies

Previous Subsection

Summary Conclusion

This Learning Module strives to provide key considerations about the process of setting goals all the way through reporting on results. It uses these considerations for a hypothetical microtransit pilot program and five common goal areas that are often considered for shared-use and mobility-on-demand projects. The paper ends with identifying a few challenges that are likely to arise in this process to help set expectations and offer ways to mitigate them. A few include Data Sharing, Data Privacy, Data Competitiveness, Data Aggregation, and Roles and Responsibilities. With many new and innovative shared mobility projects underway, it is critical to establish a measurable, data-driven approach to understand what worked well and what can be improved upon for future projects.

References/ Reading Material

• SUMC Microtransit Learning Module
• Clean Mobility Options Needs Assessment Data Collection Guide 
• FTA Mobility Performance Metrics for Integrated Mobility and Beyond Report 
• TCRP Synthesis 141: Microtransit or General Public DRT Services 
• MOD Sandbox Demonstration Evaluation Reports 
• Community Engagement Process Development Public Participation Playbook 
• Using Human Centered Design to Advance Civic Engagement in Nonprofits
• Objective Data Driven data Sharing for Transit Agencies in Mobility Partnerships 
• De Montjoye, Y. A., Hidalgo, C. A., Verleysen, M., & Blondel, V. D. (2013). Unique in the Crowd: The privacy bounds of human mobility. Scientific reports, 3, 1376