International Journal of Project Management Vol. 15, No. 2, pp.

67-72, 1997
~ ) Pergamon Copyright © 1997 Elsevier Science Ltd and IPMA
Printed in Great Britain. All rights reserved
0263-7863/97 $17.00 + 0.00

S0263-7863 (96)00003-8

Evaluation of proposals for BOT

projects

Robert L K Tiong
School of Civil and Structural Engineering, Nanyang Technological University, Singapore 639798

Jahidul Alum
School of Civil and Structural Engineering, Nanyang Technological University, Singapore

This paper is primarily concerned with the evaluation of tender proposals for build, operate
and transfer (BOT) projects particularly in the Asia-Pacific region. It presents an overview of
the current practices and techniques in the selection process, namely the NPV method, the score
system and the Kepnoe-Tregoe decision-making technique. A review of the findings from the
survey of BOT practitioners is also presented. It covers the major criteria that are commonly
used by governments in evaluating BOT proposals. The survey also covers the practicality of
the Kepnoe-Tregoe decision-making technique for evaluation of BOT proposals. Copyright ©
1997 Elsevier Science Ltd and IPMA
Keywords: BOT, RFP, evaluation, criteria, Kepnoe-Tregoe

Evaluation of tender proposals for privatized infrastructure such as bankers, developers, contractors and government
projects that are let under the build, operate and transfer agencies;
(BOT) concept involves decision making by different govern- newspaper articles, articles in business magazines,
ment officials from different departments and by their consul- journal papers and conference articles on BOT projects;
tants. Choosing the best proposal would depend on three direct correspondence with governments and promoters
elements: the quality of the definition of specific criteria, the for specific views on their actual tendering and negotia-
quality of evaluation of the available alternatives, and the tion experiences.
quality of the understanding of what these alternatives can
produce. The purpose of the evaluation and selection process The competitive selection process
is to identify what needs to be done, develop the specific
criteria for its accomplishment, and identify the advantages, The typical evaluation and selection process in a com-
disadvantages and risks involved in each of the proposals. petitive BOT tender is shown in Figure 1. The process con-
This paper describes the selection process and examines sists of pre-qualification of interested promoters, evaluation
the current evaluation practices and techniques in a few of tender proposals, pre-award negotiations and concession
countries. Responses were solicited from BOT practitioners award. This process is commonly used by governments or
to establish the major criteria that are commonly used by their agencies to serve 3 purposes:
governments in evaluating BOT proposals. The practical 1. to obtain the best deal for the public;
application of the Kepnoe-Tregoe decision-making tech- 2. to be fair to all competing promoters; and
nique in evaluating BOT proposals is also discussed. 3. to allow alternative proposals to be considered.
Competitive proposals should offer solutions to the many
Research methodology requirements of government's policies, as solicited proposals
have to comply with the tender guidelines and criteria as
The study on evaluation practices and techniques as used by specified in the RFP. Based on the study of documents on
governments or their agencies in selecting BOT proposals BOT projects, the broad requirements that are commonly
is based on documented experiences and lessons as pres- stated by the government in the RFPs can be grouped under
ented by project promoters, government officials and their macro and micro requirements.
advisors in expert seminars and conferences on BOT The macro requirements are:
projects. Information was collected from multiple sources.
These are: (a) the project must be economically feasible with benefts
to the community;
• request for proposals (RFPs) issued by governments; (b) the environmental impact study must be sound;
• interviews with professionals involved in BOT projects (c) public safety;

67
Evaluation of proposals for BOT projects: R L K Tiong

] Pre-qualification ~[
v[ Tendering

~ , ~ [ Detailed
Negotiations d

I-
\
Figure 1 Selection process in a competitive tender of BOT project

(d) the project must be socially and politically acceptable, Hong Kong and Thailand, it is not the practice to carry out
and pre-qualification of BOT promoters. In these countries, the
(e) the proposed development must solve the identified invitation to tender for a BOT project is announced in the
needs for the project, represent the best solution to that newspapers and interested promoters are generally given a
need and be consistent with a co-ordinated development period of 3--4 months to prepare detailed submissions. The
strategy of government. reasons for these governments following this approach is
that they believe that the scale of investments required for
The micro requirements are: BOT projects and the keen competition will deter small
companies from submitting proposals. Instead, it will
(a) the winning consortium must be technically strong and attract only serious promoters who are financially strong.
financially sound;
(b) the technical proposal must provide the solution to meet
the macro objectives and the demand for the services Evaluation of proposals
adequately; and The term 'evaluation' describes the procedure for the
(c) the financial proposal must be attractive and com- assessment of tender proposals submitted by pre-qualified
petitive to support the level of tolls or tariff specified or promoters. For BOT projects, the selected promoters
proposed. would prepare schemes and financial schedules of financing
and operating revenues based on the government require-
ments for the physical structure, the methods of payment
Pre-qualification and their sources of funds for the project debt and equity.
Like the traditional public sector construction contracts, In evaluating the proposals, governments usually follow
some form of pre-qualification is adopted for most BOT the Chadwick/Demsetz principle 1 which states that a con-
projects in countries such as Australia, Canada, the cession contract must be awarded to the promoter offering
Philippines, the UK and the US. The main aim of the the lowest consumer price, or alternatively, to the one
request of qualification (RFQ) is to shortlist a number of offering the price/quality combination which is judged
competitive proposals by consortia which consist of reputable superior to all others. For the selection of the best proposal,
and experienced contractors, operators and bankers. In a the offer price on tolls or tariff would therefore be the
competitive tender for BOT concession, the number of predominant, though not necessarily the only, criterion.
promoters for the BOT contract may be limited due to the Other criteria would include cost-related factors (e.g. the
technical, financial, and/or sometimes political constraints construction cost and the operation and maintenance costs),
imposed upon the promoters. price-related factors (e.g. the extent of future toll in-
Bruce Nicholls, Executive Director, Major Projects and creases), technical factors (e.g. the reliability and long-term
Infrastructure for New South Wales State Government, maintainability of the plant facility) and environmental
Australia, in correspondence with the author on this topic factors.
stated: "As a general practice, detailed submissions are not For most projects, the promoters, after submission of
sought in initial responses to an invitation to develop a proposals, undergo several rounds of negotiations before
major infrastructure project. The preferred option is to the decision is made and the concession is awarded. This
identify the characteristics of the companies or consortia sub- is the process whereby some will be eliminated, others will
mitting and from this develop a 'shortlist' of perhaps three drop out on their own while two to three are shortlisted for
or four bidders. This procedure ensures that unsuccessful final negotiations. This happened for BOT projects in
bidders are not required to incur unnecessary costs". The Australia, Hong Kong, Indonesia, the Philippines, the UK
pre-qualification process will therefore ascertain the financial, and the US.
technical and managerial ability of each consortium in For countries which do not practise pre-qualification such
undertaking the project. as Malaysia and Thailand, the process is different. In
For governments in some countries such as Malaysia, Malaysia, a promoter is selected, based on evaluation of the

68
 Evaluation of proposals for BOT projects: R L K Tiong

proposals which should contain detailed submissions. The make their final decision. For example, in Hungary, the
promoter is then issued with a letter of intent which invites government will fix the level of tolls and the competition
the promoter to enter into an exclusive period of negotiations will be based on the lowest concession period offered. For
with the government. The concession is only awarded after the Canadian Northumberland Strait Crossing, the annual
the government is satisfied with the best and final offer from subsidy to be paid by the government to the successful
the promoter. Successful negotiation therefore plays an winner was the final and most important criterion.
important role in the selection process.
In Thailand, the government agency will hold pre-award The Kepnoe-Tregoe decision-making technique
negotiations with the shortlisted promoters and post- Several governments, such as the Hong Kong government
award negotiation with the selected promoter. If the post- and the New South Wales State government, have used
award negotiations are successful, the government agency more sophisticated evaluation techniques, particularly the
will recommend to the Cabinet for the final approval. In Kepnoe-Tregoe decision-making technique in the analysis
both countries, the selected promoters are informed that if and evaluation of BOT proposals. The Kepnoe-Tregoe
the negotiations are not satisfactory, the government will decision analysis method 2 is used by some commercial
choose the second ranked promoter to continue the selec- companies in evaluating alternative proposals for a new
tion process. The government always keeps such options investment. The major elements in this technique consist of
open. the evaluation statement, the MUST criteria, the WANT
criteria and the evaluation matrix. The modified procedures
Alternative proposals of this technique for evaluation of BOT proposals are
A high degree of flexibility is usually contained within the shown in Figure 2 and are explained in the following
BOT system of tendering as governments generally permit sections.
alternative proposals. Alternative proposals are usually The validity of the modified procedures for BOT projects
required to be accompanied by a conforming proposal in was tested by seeking responses of the government officials
order that the government may assess the costs and benefits on the practicality of its major elements through an opinion
of the alternatives. survey. In order to improve the quality of responses to the
questions, the survey was targeted at government officials
and their advisors. These persons were asked to respond to
Evaluation criteria and techniques the questions with reference to a specific BOT project
The current techniques of evaluation of BOT proposals are that they were involved in evaluating. Out of 75 question-
based on the net present value (NPV) method, the scoring naires mailed, 30 government officials and their advisors
system and the Kepnoe-Tregoe decision-making technique. responded, giving a response rate of 40%. Respondents
who declined to participate cited undertakings of confiden-
The NPV method tiality of tender information. Several cited sensitivity of the
information as their BOT projects are still under negotiation
Some governments evaluate the commercial and financial
and disclosure of their previous experiences in the projects
package by performing an NPV calculation to discount the
may undermine the success in the final selection.
project cash flows due to the promoters. The lower the
NPV, the cheaper the offer. For utility projects, the com-
parison is straightforward as it is generally based on govern- The evaluation statement. In this technique, the evaluation
and selection process must first begin with the adoption of
ment's offtake agreement. For a tolled road, it is more
an evaluation statement. The evaluation statement, once
complicated as traffic is not normally guaranteed. Never-
agreed by the selection committee, provides the focus for
theless, as long as there are adequate traffic studies and
all the evaluation and negotiations that follow and sets the
conservative traffic forecasts, the government will compare
limits of the choice.
the NPV of the cash flows based on the toll revenues,
operation and maintenance costs, financing charges and Government responses. In order to establish the validity
and usefulness of the evaluation statement, the government
loan repayments.
The advantage of using the NPV method is that the respondents were asked whether they agreed or disagreed
that the following statements would have been useful during
proposals could be compared based on calculated numbers.
The method, however, ignores the relative advantages the evaluation of the BOT projects that they were involved
and disadvantages of the technical solution in different in. These statements are commonly stated in the RFPs as
proposals. governments' objective in inviting tenders.
These statements are:
Score system (i) "Select the proposal that offers the best overall value
In this system, points are given to the selection criteria and for money".
the proposals containing the financial package, technical (ii) "Select the proposal that offers the most attractive
designs and others are evaluated based on the scores obtained. financial package and most cost-effective technical
The proposal with the highest score is considered to be the solution".
best overall proposal. The advantage of this method is that (iii) "Select the proposal that is best researched overall in
several criteria are used in comparing the proposals. The the technical and financial aspects of the project".
NPV technique could also be used first in evaluating the
cash flows. The disadvantage of the score system is that it The responses are shown in Table 1. The most favoured
assumes that all the criteria are of equal importance. statement is statement no. 2. The positive responses show
The NPV and the score system are the two most common that the respondents agreed on the practicality of the
methods used by governments. Some governments use statements. Respondents perceived that the governments
different methods of evaluation and some may choose a want a package that is financially attractive and technically
single criterion based on their objectives and priorities to cost-effective. A few other evaluation statements were

69
Evaluation of proposals for BOT projects: R L K Tiong

Criteria for evaluation. Once the evaluation statement is

I Evaluation stated for the areas in which the decision is to be made, the
Statement
criteria are to be established. For the New South Wales
government, the criteria adopted for determining a shortlist
I MUST is flexible and may vary according to the particular needs
Criteria of a project. Criteria that could be adopted include the
bidder's previous experience - 'track record' - in projects
relevant to the one under consideration; the range of exper-
tise and professional competence that can be found within
Criteria the organization concerned; financial and technical strength,
including access to lines of credit, within the project, and
any uniquely attractive features in the respective bids. For
the Kepnoe-Tregoe technique, the criteria can be divided
Evaluation into two categories: MUSTs and WANTs 2.
Matrix &
Analysis The MUST criteria. The MUST criteria are mandatory:
they must be achieved to guarantee a successful decision
* criteria
* weights and any proposal that cannot fulfil a MUST criterion will
* SCOreS be discarded.
* sensitivity Government responses. Through a study of the RFPs, the
analysis author found that there are three common criteria that fit the
description of MUST criteria. In the survey, the gov-
ernment respondents were asked whether these three criteria
could be accepted as MUST criteria for selection of pro-

i Aust 1
Scores
posals in their BOT projects. They were also asked what
other criteria should be included. The three MUST criteria
postulated are:
Clarification
& levelling
of proposals (i) Proposals must be complete and must comply with the
tender guidelines.
Proposals (ii) Promoters must have proven capacity (financial and
technical) and experience in construction and
Negotiations operation of similar project.
with promoters (iii) Promoters must have a local company in its team.

Table 2 shows that while the majority of the respondents

proposal
agreed with criteria 1 and 2, most did not agree with
criterion 3. The common reason given was that the inclusion
Figure 2 Selection of BOT proposals on the Kepnoe-Tregoe of a local company is not essential at this stage of evalu-
technique ation, but it will be an important requirement prior to the
implementation of the project.
Table 1 Responses on evaluation statement
Other evaluation criteria that the respondents felt are
Evaluation Responses by governments relevant and could be included as MUST criteria in evaluating
statement their BOT projects are:
Yes No
Statement 1 20 10 (i) NSW toll roads, Australia
Statement 2 22 8 • proposals must address distribution of windfall
Statement 3 17 13 profits arising from unforeseen development or
other cause
(ii) Canada's Northumberland Straits Crossing project
added in by the respondents for specific BOT projects. This
• promoters must comply with environmental
shows that the evaluation statement is acceptable to the
requirements
respondents and is therefore useful in providing the focus
• promoters must meet requirements of the financial
in the evaluation process of a BOT project. Some of these
package
statements are as follows:
• promoters must meet other requirements regarding
(i) Sydney water treatment plants local involvement such as labour, materials purchases
• proposed plants must be capable of being expanded etc.
or upgraded in time at an affordable price
(ii) Sydney airport rail link
Table 2 Responses on MUST criteria
• the evaluation statement must also include exter-
nalities such as road congestion relief MUST Responses by governments
(iii) Light Rail Transit system, Philippines criteria
Yes No
• proposed project must be financially viable
(iv) California transportation projects 1 26 4
• projects must make good transportation, environ- 2 27 3
3 I0 20
mental and business sense.

70
 Evaluation of proposals for BOT projects: R L K Tiong

(iii) Malaysia's power plants Table 3 Government responses on W A N T criteria

• proposals must meet environmental requirements WANT Responses by governments
(iv) California transportation projects criteria
• project proposals must make good transportation Yes No
and business sense 1 29 1
• project must have local support 2 24 6
• project must not have serious environmental problems 3 29 1
(v) Texas High Speed Rail project 4 27 3
5 23 7
For this project, promoters must address the following
issues in their proposals:
• local procurement of materials the various risks such as future changes in taxation
• minority business opportunities and future changes in property enhancement etc.
• maximum attention to environmental concerns (ii) Victoria "s Loy Yang B power plant, Australia
• strategic economic development opportunities • parent company support
• maximum public input/public meetings • ability to close transaction according to timetable
(iii) Tate's Cairn Tunnel, Hong Kong
The WANT criteria. After the MUST criteria are estab- • ability of government to control/regulate the level
lished, the other criteria are categorized as WANTs. of toll charges
According to this technique, the proposals would be judged (iv) 2nd Severn Bridge crossing, UK
on their relative performance against a set of WANT • realistic project construction schedule
criteria, not on whether or not they fulfil them. The function (v) Skye Bridge crossing, UK
of these criteria therefore is to give the assessors a com- • whole life cycle cost of the proposal
parative picture of proposals - a sense of how the proposals • maintainability and durability of the design
perform relative to each other and which could perform (vi) California's transportation projects, US
best. • ease of implementation
Each MUST and WANT criterion could also be sub- • technical innovation
divided into its own set of sub-criteria. For example, in the • degree of support for achieving civil rights objectives
evaluation of the Hong Kong BOT tunnels, the sub-criteria regarding the utilization of minority and women
for the financial package included toll structure, mechanism business enterprises
for toll adjustment, debt/equity ratio, immunity against (vii) Texas High Speed Rail Project, US
market risks, drawdown of loans and debt repayment • degree of public convenience and necessity of project
schedules, guarantees and undertakings, royalty and profit-
sharing mechanism, protection against uncertainties such Weighting the criteria. Once the WANT criteria have been
as interest rates and foreign exchange variations. The identified, each one is weighted by the Selection Committee
MUSTs therefore enable the governments to decide who according to its relative importance in achieving the govern-
gets shortlisted whereas the WANTs lead to the award of ment's objective of, say, a low and stable toll structure. The
the concession. most important criterion would be identified and given a
Government responses. Through a study of the RFPs, the weight of 10. All other criteria would then be weighted in
author found that the criteria commonly used by govern- comparison with the first, from 10 (equally important)
ments in evaluating BOT proposals could be grouped into down to a possible 1 (not important). No attempt should be
five main criteria. The five criteria are: made to rank the criteria.

(i) Degree of attractiveness of financial package. Evaluation of proposals against MUST criteria. In this
(ii) Financial returns to government and benefits to evaluation, a proposal either meets all the MUST criteria
community. or it does not. As shown in Table 4, if it does not, it will
(iii) Relative soundness of technical solution for project be given a NO GO and is immediately excluded. In actual
implementation. practice, the government will hold a meeting with the
(iv) Relative experience and expertise of the project promoter promoter to explain the deficiencies of the proposal and it
in similar projects. is up to the promoter to make good the deficiencies.
(v) Degree of environmental impact.
Evaluation of proposals against WANT criteria. At this
The government respondents were given these five criteria stage of evaluation, all criteria have been made visible and
and were asked whether the five criteria could be accepted the WANTs have been weighted. The proposals will be
as WANT criteria and used in evaluating the BOT projects examined and the relative advantages of each proposal
that they were involved in. They were also asked to suggest judged. As each promoter is evaluated on the basis of all
other WANT criteria. the WANT criteria, its overall performance and ability to
The results are shown in Table 3. produce desirable results will become clear. Table 5 shows
The majority of the respondents therefore agreed with the how the evaluation of the relative performance of the
five WANT criteria. Other criteria suggested by them for
their BOT projects were as follows: Table 4 Evaluating proposals against MUST criteria
(a) (b) (c) (d) (e)
(i) Sydney airport's rail link, Australia
• planning and development issues - the estimated MUST criteria Proposal A Go/No go Proposal B Go/No go
effect of new transport infrastructure on the dev- In Columns (b) and (d), the government would give the reasons as to
elopment of nearby property why the proposal has passed or failed the MUST criteria. By listing this
• a comprehensive risk analysis on who would bear information, the process has become visible and transparent.

71
Evaluation o f proposals f o r B O T projects: R L K Tiong

Table 5 Evaluating proposals against WANT criteria 2 Kepnoe, C H and Tregoe, B B The New Rational Manager Princeton
Research Press, Princeton, NJ, US (1981)
(a) (b) (c) (d) (e)
WANT criteria Weight Proposal A Raw score Weighted
information score Bibliography

From the above table, the questions that need to be answered are: How 1 Anderson, G 'Lessons from regional experiences' Conference on New
does each proposal perform across the board? How does it measure up Opportunities and Issues in BOT Projects Institute for International
against the other proposals on total performance? The questions can be Research, Jakarta, Indonesia (1989)
answered by computing and examining the total raw score and the total 2 Attajarusit, T 'The Thai Government's perspective on BOT projects'
weighted score of each proposal. A weighted score is the raw score of a The Asian Conference on Planning, Packaging & Implementing BOT
proposal multiplied by the weight of the criterion to which the score refers. Projects Singapore (1988)
The total weighted score of each proposal functions as visible comparative 3 Guidelines for Conceptual Project Proposals for Toll Revenue Trans-
measurement of the alternatives. portation Projects California Department of Transportation, USA
(1990)
4 GuidelinesforPrivateSectorParticipationinlnfrastructureProvision
proposals scored against all the WANT criteria can be New South Wales, Australia (1990)
done. 5 Invitation To Promoters for the Development, Financing, Construction
and Operation of a Channel Fixed Link Between France and the UK
Sensitivity tests. Where appropriate, sensitivity tests can be Department of Transport, UK (1985)
6 Invitation To Promoters for Development, Financing, Construction
applied to the evaluation matrix to ensure that the result and Operation of a Third River Crossing at Dartford Department of
would be sufficiently robust and would not be easily altered Transport, UK (1986)
due to a slight variation in one or more of the weighting 7 Porter, J E and Matson, C R 'A franchized tollway - the design,
factors. financing and management of the Tate's Cairn Tunnel, Hong Kong'
Conference on Tunnelling Institution of Engineers, Sydney, Australia
(1990)
The tentative choice. The total weighted score would give 8 Project Brief for Eastern Harbour Crossing, Hong Kong Highways
the government a tool for selecting a tentative choice. Department, Hong Kong (1986)
Although the tentative choice may graduate to the status of 9 Proposal For Guangzhou-Shenzhen-Zuhai Superhighway Gordon Wu,
final choice, governments are often very cautious and Hopewell Holdings, Hong Kong (1983)
10 Revised Terms of Reference for Submission of Investment for Mass
would assess all the risks involved. They would hold Rapid Transit System in Bangkok, Part 1, Stage 1 Expressway and
further negotiations with the promoter to obtain a better Rapid Transit Authority, Thailand (1987)
deal and to ensure that a security structure of contracts is
in place before the concession is awarded.

Conclusion Robert Tiong graduated from the

University of Glasgow, UK with
The evaluation and selection process in a BOT tender was first class honours in civil engineer-
discussed in this paper. The current techniques used in ing. He is a professional engineer
in Singapore, holding a master's
evaluation of BOT proposals are based on the net present degree in construction management
value method, the score system and the Kepnoe-Tregoe from the University of California,
decision-making technique. This paper concludes that Berkeley and a PhD from Nanyang
government's evaluation goal should be to select a balanced Technological University. He is
currently a Senior Lecturer at NTU
proposal that is financially attractive and technically cost-
and the Coordinator for the MSc
effective. In the final selection, the following criteria are Programme in International Con-
commonly used: struction Management. He teaches
international project financing to
n degree of attractiveness of financial package; the MSc students as well as to PRC construction professionals under
• financial returns to government and benefits to the a World Bank sponsored training project. His current research
interests are in international project financing and privatization of
community; infrastructure projects with special emphasis on BOT projects. He has
• relative soundness of technical solution for project published extensively on the subject including a monograph on 'The
implementation; Structuring of BOT Projects" and a paper on "Critical Success Factors
• relative experience and expertise of the project promoter in Winning BOT Contracts '.
in similar projects;
• degree of environmental impact.

Indeed a better understanding of the evaluation and selec-

tion process and the criteria used by governments will Professor Alum is an Associate Professor in the School of Civil and
definitely help the promoter to respond appropriately and Structural Engineering, Nanyang Technological University, Singapore
positively in the tendering and negotiation process. and is former Director of the Centre for Advanced Construction Studies
in NTU. He has more than 35 years of practical and academic
experience in his areas of specialization of construction technology and
References project management. He has contributed many papers to various
international conferences and journals and is actively involved in
1 Demsetz, H "Why regulate utilities?' Journal of Law and Economics research and consultancy related to his areas of specialization.
(1968) 55-66

72