Clearinghouse
TMIP Newsletter Archive
TMIPConnection
The Travel Model Improvement Program Newsletter
Issue 29, Spring 2008 (PDF format 
)
FHWA-HEP-08-014
The Travel Model Improvement Program is sponsored by:
U.S. Department of Transportation
Federal Highway Administration
Federal Transit Administration
Commentary on TRB Special Report 288
One of the most important recent documents in the travel forecasting industry has been published, and while it confirms about the practice what most modelers suspected from their more limited, personal view of their own travel models, it is also rife with recommendations and strategies for improvement.
TRB Special Report 288, Metropolitan Travel Forecasting: Current Practice and Future Direction, examines metropolitan travel forecasting models that provide public officials with information to make decisions on major transportation system investments and policies. The report explores what improvements may be needed to the models and how federal, state, and local agencies can achieve them.
TRB Special Report 288 was sponsored by USDOT Office of the Secretary, FHWA and FTA. Its purpose was to determine the state of the practice in metropolitan travel forecasting, identify technical shortcomings of the models for their intended uses and recommend actions needed to ensure appropriate technical processes are being used. This issue of TMIP Connection features two commentaries on SR 288: one is from Martin Wachs, chair of the TRB SR 288 committee, who discusses the report and its implications from the committee's perspective. The other is from FHWA Associate Administrator of Planning Environment and Realty Gloria Shepherd, who gives the FHWA perspective on and response to SR 288.
Martin Wachs: Commentary on TRB Special Report 288
The committee included practitioners from MPOs and state DOTs, academics and consultants, supported by a technical advisory group of travel model consultants, with invaluable assistance from TRB staff.
Information for the report was gathered from multiple sources: an extensive MPO survey, in-depth interviews with selected MPOs, literature review and briefings from stakeholders. The collective expertise of the committee was employed, as well. The report was reviewed by yet more practitioners, academics and consultants, and I feel it is safe to say that by the time the report was released, certainly every element of the metropolitan travel forecasting community had contributed to or reviewed the report.
According to the committee that produced the report, travel forecasting models in current use are not adequate for many of today's necessary planning and regulatory uses.
FINDINGS
Models
We learned that while the basic approach, unchanged for 50 years, is the 4-step model, there are many variations in the complexity of approach and there is no single correct approach to travel forecasting that would work for all MPOs. Complex issues have led to more complex models, and some larger MPOs are adopting more advanced approaches. These models, where implemented, appear to work well.
The shortcomings of the current 4-step models are that, because they are not behavioral in nature, they are inadequate for many applications of travel demand analysis, such as:
- Induced travel
- Land use policies
- HOT lanes and time variable road pricing
- Environmental justice assessments
- Telecommuting
- Mode of access to transit
- Traveler response to congested networks
Furthermore, certain modes are poorly characterized, such as non-motorized travel and freight and commercial vehicles.
Models are also inadequate for supply-side analysis. They yield inadequate disaggregate estimates of volumes and speeds on specific routes by time of day. These shortcomings contribute to difficulty in:
- Evaluating traffic operations movements
- Analyzing time shifting in congested networks
- Evaluating freight movement policies
- Analyzing emissions estimates
- Planning evacuations
Practice
The travel forecasting practice has significant shortcomings. The primary problems are inadequate data, optimism biases, quality control issues and validation errors. The obstacles to advancing the state of the practice include:
- Preoccupation with meeting immediate demands of production – doing the current work
- Fear of legal challenges
- Significant budget and staff limitations
- Insufficient evidence that advanced models can be implemented for a reasonable cost or provide significant improvements
- Poor or inadequate data
Federal Role
There is the added difficulty that federal support for models has not been commensurate with the demands on modeling. Funding for travel forecasting research reached levels of $15 million per year some 30 years ago but has recently fallen to about $2.5 million today. Despite this, today we make ever more stringent demands on model practice and accuracy.
The FTA is to be commended for its efforts to address quality control, but greater funding support is needed.
RECOMMENDATIONS
SR 288 makes a number of recommendations for improving the state of metropolitan travel forecasting. Of these recommendations, some are specific to MPOs, some to states, some to the federal government and some apply to all.
MPOs would benefit from an established national cooperative research and development program. The program should be funded at $4 to $5 million annually, and should be governed by the MPOs. The New York State Association of MPOs shared-cost initiative program is a model for this proposed program. MPOs should continue to perform and participate in peer review, to develop partnerships with universities that promote innovation and experimentation in modeling, should perform reasonableness checks of project forecasts and should rigorously document experience with advanced practice.
For their part, states should support the development of an MPO cooperative research program, they should cooperate with the MPOs to evaluate socio-economic forecasts used for modeling and they should continue to develop and support regional area Model User Groups.
The federal government continues to play a large part in urban model quality. SR 288 recommends that the federal government:
- Support and provide funding for incremental improvements to 4-step models that are appropriate for use
- Support and provide funding for development, implementation and evaluation of advanced models
- Continue TMIP
- Increase funding – 0.005% of federal aid is about $20 million, which is roughly comparable to the $15 million level of support that prevailed in the 1960s and 1970s
- Perform MPO Certification of models based on a checklist, incorporating MPO peer reviews
- Provide flexibility for MPOs to apply models appropriate to their needs
- Continue TRANSIMS deployment
Federal, state and local collaboration is essential as well. Together we should establish goals, responsibilities, improved training elements and the means of improving travel models—perhaps through a steering committee of principal stakeholders. We also should develop a national handbook of practice and keep it current—perhaps through a national organization that could bring partners together and perhaps funded by the nascent MPO Cooperative Research Program, NCHRP and the federal government. Together we should document data requirements for updating travel models, validating models, developing freight models and meeting air quality conformity requirements.
There is resistance to change, yet a strategy for change is crucial. For 40 years advances in research and development have led to only incremental change. Today we must break this cycle and harness the resources of each level of government. We can return to the creativity and innovation of the early days of travel forecasting.
Gloria Shepherd: Commentary on TRB Special Report 288
SR 288 tells us that the state of the practice of travel forecasting needs significant improvement. Here we highlight three major findings from the report; two criticisms and an observation:
- There are problems with data and quality control:
- issue for both current 4-step models and advanced models
- There are fundamental shortcomings in the current models:
- highly aggregate
- lack of time dimensions
- No single approach is correct for all applications:
- Approach depends on the nature of the question and the types of analyses that are conducted
FHWA, on the whole, concurs with the findings in SR 288 and has a plan that includes ongoing, immediate and long-term recommendations to address the issues raised in the report.
On an ongoing basis, FHWA will continue TMIP and TRANSIMS deployment, and continue working with our key partners on travel modeling issues through the TMIP program.
FHWA has several immediate actions to implement that promote good travel model practice. For example, TMIP has begun a web series addressing current travel modeling and its practice issues; and an exemplary MPO project, which aims to illustrate the better travel modeling practices from some of the MPOs nationwide. FHWA has increased TMIP technical support with more peer reviews and focused topical peer exchanges, continuation of the recently implemented monthly technical webinars, exploration of methods for greenhouse gas estimation and EPA's MOVES applications, and support for the Census Transportation Planning Package (CTTP).
Furthermore, I am bringing the FHWA Head Quarters modeling function together into one group. This will make us perform more efficiently and with greater economies of scale. In the immediate term, additional Surface Transportation Environment and Planning Cooperative Research Program (STEP) funding has been directed to travel modeling.
In the long term, my office will work to include resources to improve the state of the practice in our draft reauthorization proposal. We will assist in the creation of a National Travel Demand Forecasting Steering Committee, as recommended in the report, and support the committee in the development of a handbook of practice.
FHWA recognizes the need to both improve the travel modeling practice itself and develop new methods. We recognize that development of new travel analysis methods will take significant resources of time, talent and funding, and we are committed to supporting them.
Boise, Idaho, Peer Review
By documenting and publishing peer review results, the TMIP peer review program is useful to the host agency, participants involved in the peer review and to the community at large.
The Community Planning Association of Southwest Idaho (COMPASS), the Boise MPO, was the site of a TMIP peer review June 5-6, 2007. The review panel members commended COMPASS staff for their integrity, dedication and knowledge of the travel modeling process and found the Boise model sound.
The peer review report focused on exploring specific technical issues and recommendations. The Boise peer review panel identified eight issues with the model itself and made several recommendations for overall model improvement. The issues ranged in subject matter from demographic data, to trip generation and distribution, to mode choice, to assignment. Additional recommendations regarded commercial vehicle trips, external trips, and post processing of assignment speeds. To read a detailed report of the issues, recommendations and model documentation of this peer review, please go to: http://tmip.fhwa.dot.gov/services/peer_review_program/documents/compass/ or to: http://tmip.fhwa.dot.gov/services/peer_review_program/status.stm to read other peer review reports.
Web Knowledge and Information Exchange – Update
Are you part of the latest rage? Once a month your colleagues meet online to discuss topics of technical interest to the modeling community. TMIP has started a web-delivered series of knowledge and information exchanges (WKIE), and we're told they are well received and helpful. The WKIEs are held once a month. Past topics have included:
- The Census Transportation Planning Package (CTPP)
- Land Use Modeling
- National Household Travel Survey Data
- Modeling Activities at MPOs
- Activity-Based Modeling and TRANSIMS
- TRB Highlights
- Shining a Light inside the Black Box
- Model Series
- Motivations and Data
- Model Testing
- Transportation Supply and Travel Distribution
- Translating Results into Insights
The exchanges started out with 80 access lines but were quickly bumped up to 120 to accommodate the high demand. Delivering live seminars over the web is a low-cost method to disseminate information to a broad contingent of our community. Recordings of some of the WKIEs are archived so you can access them at a later date, but nothing beats being able to post your question live! To view past WKIEs and register for future ones, visit the TMIP website at: tmip.fhwa.dot.gov/discussions/webinars/.
Hot Topics – Correlation Coefficient vs. Coefficient of Determination
By Rich Arnold, P.E., Senior Transportation Analyst – Oregon Department of Transportation
R vs. R2
Oregon Department of Transportation (ODOT) places great emphasis on clear and concise model development documentation, including a more detailed technical explanation of the model analysis than most modeling agencies.
In an attempt to evaluate some types of national trends in acceptable model documentation (standards do not exist), I perused several hundred model development reports. The primary goal of this investigation was to evaluate how other jurisdictions were reporting their model development, and to help identify and evaluate reasonable model performance indicators. Unfortunately, SR 288, "Metropolitan Travel Forecasting: Current Practice and Future Direction," and its supporting technical report were not available.
In reviewing the numerous reports, I discovered two model calibration documents that were frequently referenced by most reports: Calibration and Adjustment of System Planning Models and Model Validation and Reasonableness Checking Manual.
I found discrepancies in the way several model performance measures were being reported by the various jurisdictions. In talking through the inconsistencies within our office, it was suggested that I submit a question to the TMIP Listserv for modeling insight, which is excerpted below. The result of doing so seems to have stirred up quite a discussion on the topic that I introduced; about half of the responses came from academia.
In part, I asked "What is the acceptable target, R or R2? It is a little confusing since 'correlation coefficient' and 'coefficient of determination' are actually two different values and have different statistical significance. It appears that they are mistakenly being used interchangeably to report model accuracy. I would expect the R2 value to be more significant for reporting model performance, which is what most software packages seem to produce. Can anyone help me understand this?"
Mr. Conder's response "…The short and unimportant answer is R2…" and Dr. Ortuzar's response "…You seem to be talking of an extremely simple model, with only one variable … in that case R or R2 is probably the same…" seem to come closest to what I was asking; where Ortuzar implies that R or R2 shouldn't matter for this type of model, and Conder suggests that the measure is unimportant.
Herein lies the problem, Mr. Ismart's piece clearly uses the term "correlation coefficient" (R) and states that it should typically begreater than 0.88 (note that he never calls this a standard). Numerous model reports reference Ismart's work, but substitute the term "coefficient of determination" (R2) for R, while still using the measure value 0.88 as the datum; some actually go a step further by stating the FHWA standard is R2 = 0.88, again referencing Ismart's work.
Is this simply a technical oversight, such that the modeling world assumes that technical people will know the difference and "not sweat the small stuff?" Is the inconsistency defined here simply ignorance of basic statistics? Are we misusing statistical measures? Should it even matter? Lawyers make a living using words and understanding nuances in language. They may not know the difference between R and R2, but if they can suggest a weakness in the overall model assessment due to the misreporting of statistical measures, they can create reasonable doubt about a modeler's "expert judgment." This makes it easy to get caught up in the game of "my expert witness" is better than "your expert witness." In Oregon, this issue is faced more and more, requiring the modeling community to be more rigorous in documenting what is done and why.
The general TMIP discussion on this topic was of little help for my immediate inquiry. Most of the discussion centered on theoretical concepts, whereas I was hoping to get responses from people who develop travel demand models and apply them to planning projects, and who deal with the same validation and documentation issues at the local jurisdiction level. In almost all cases, the discussion focused on minor issues and generally missed the purpose of the posting. Several responses seem to imply that both R and R2 are really insignificant measures, and the general feeling was that "expert judgment" is most important (which I agree is essential for model assessment). The question was never about defining and understanding statistical variables; there are tons of books and websites devoted to just that purpose. Rather, the intent was to focus in on the usage of statistical measures in a correct and consistent manner for modeling; in this case just one of many functional performance measures was discussed. Based on the response from this posting, the TMIP discussion revealed to me a lack of information available for the everyday modeling practitioner. This caused me to rethink my approach to documentation, such that I have included additional verbiage to clearly and concisely explain the various statistical performance measures used as a basis for my professional judgment.
To see the full text of this discussion on the Listserv, or to post your own query or peruse the archives, go to: http://listserv.tamu.edu/archives/tmip-l.html.
Transit Travel Forecasters' Corner
By Ken Cervenka, FTA
You may have noted my column is now called Transit Travel Forecasters' Corner rather than Transit Modelers' Corner. This change emphasizes the point that the purpose of modeling is not simply to build models, but to use models as a tool to help with the preparation of meaningful forecasts for use in informed decision-making.
The winter 2008 issue of TMIP Connection contained an overview of the "FTA Travel Forecasting for New Starts" workshop held in St. Louis, Missouri, on September 19-20, 2007. One of the highlighted items was transit rider data for model testing and before-and-after studies. The critical importance of data as the foundation for good model development is stressed in the June 4, 2007, Federal Register with this statement: "FTA adopts as final the proposal—for implementation in May 2009—that travel forecasts for both New and Small Starts submitted in support of a request to enter preliminary engineering (PE) or project development (PD) be based on travel models that have been validated against data sufficient to describe current ridership patterns." Projects with forecasts that do not meet this condition will not be allowed to advance. To support compliance with this requirement, last year FTA awarded about $12 million from the alternatives analysis discretionary program (Section 5339) funds.
FTA encourages agencies to contact staff in its Office of Planning Methods (where I work) so that we may share some of the insights we have gained in the past few years in our efforts to improve transit on-board surveys. Ideally we'd like to work with you on the statement of work for the endeavor. Anyone who has developed these surveys and the methods to collect them is familiar with difficulties of reasonably representing the myriad of disparate users of the system and creating questions that respondents understand. Recent improvements that have been made include avoiding the reporting of round trips when one-way trips are being requested; collection of information about the trip's origin and destination and the actual transit paths used; and use of supplemental data to expand the main rider survey in such a way to significantly reduce non-response biases. To facilitate this sharing of knowledge, FTA has recently placed Guidance on Transit Rider Surveys on the web: http://www.fta.dot.gov/planning/newstarts/planning_environment_7275.html.
The guidance will be updated as needed to reflect the latest information available on best practices and should be worth reviewing by any agency desiring to get information about transit riders. There are key questions to consider as early in the planning process as possible:
- What are the purposes of a survey? In addition to understanding travel patterns and rider characteristics, is it for updating and validating a regional travel model? Or is the survey focused on a particular corridor and/or transit markets within that region? While addressing multiple purposes (such as customer satisfaction surveys) sounds efficient, it results in a lot more questions on a form. Too many questions will lower the response rate and/or quality of responses to the core questions of modeling interest.
- How much time and money are available for the data collection program—and are amounts sufficient to meet the intended purposes? This includes not just fielding the survey, but performing other activities related to preparing the sampling plan, questionnaire design, pilot tests, supple-mental surveys/counts to control the expansion process, geocoding, cleanups and data analysis.
- How large a sample is needed, what data items will be collected and how will the collected data be used?
- What approaches will be used to correct for the expected non-response biases?
- What survey methods (self-administered versus interviewed) will be used for the main and supplemental surveys?
So if you are planning to conduct a transit rider survey, take a look at the referenced web document to see if it may help you answer these questions. And feel free to contact me at Kenneth.Cervenka@dot.gov (202-493-0512) for additional consultative advice related to your specific needs—regardless of whether you expect to pursue federal funding.
And finally: let me know if you have any ideas about future transit forecasting topics you would like to see in this column—or that you would be willing to write. Bye for now!
TRANSIMS Status
Two TRANSIMS case studies are well underway. The first, led by a team at Louisiana State University, is modeling multimodal evacuation scenarios for New Orleans, Louisiana. The second, led by a team at Resource Systems Group, is modeling Transportation Systems Management and interchange scenarios for Burlington, Vermont. Three new case studies are kicking off this year in Atlanta, Georgia (GRTA); Minneapolis, Minnesota (Citilabs) and Des Moines, Iowa (Cambridge Systematics). Topical research kicking off this year includes Pricing (VA Tech), Travel Time Reliability (GA Tech), Visualization Tools (Balfour) and ITS (Cognometrics). For more information on TRANSIMS projects, please visit the project zoo in the TRANSIMS forums by going to: http://transims-opensource.org/phpbb/index.php and clicking on Project Zoo.
The TRANSIMS open source community, which numbers over 140 registered users, is enjoying an excellent website that includes many downloadables and interactive parts. There are a thriving forum and a recently established wiki site. New bug tracker software is being implemented. The community is seeking to elect its own leaders and managers and looking forward to taking off as a grass roots effort. For more information, or to join this community, go to: http://transims-opensource.net/.
Upcoming Events
TMIP Calendar
May 9, 2008
Peer Exchange on Travel Model Validation
Washington, DC
May 12-15, 2008
Introduction to Urban Transportation Planning
Richmond, CA
May 13, 2008
Shining a Light Inside the Black Box: Translating Results Into Insights for Decision Makers
Online Webinar
June 2-6, 2008
Introduction to Urban Transportation Planning
Baton Rouge, LA
June 12, 2008
Household Travel Survey: Lessons Learned
Online Webinar
June 19-20, 2008
5th Oregon Symposium on Integrated Land Use-Transport Models
Portland, OR
June 22-24, 2008
TRB Innovations in Transportation Modeling (ITM)
Portland, OR
TMIP wishes to express its thanks to all the members of the travel model community that step up and participate in our many projects. Without the voluntary support and cooperation of these planners and modelers, TMIP would not be the program that it is today. We rely on you, and thank you.
To subscribe to this free newsletter, unsubscribe, or change your mailing address, please send a detailed email to: tmip@tamu.edu. Put "TMIP Connection" in the subject.
