TRANSPORT RESEARCH LABORATORY TITLE by Economic assessment of road projects: do our current procedures tell us what we want to know? J Hine and M Cundill Overseas Centre Transport Research Laboratory Crowthorne Berkshire United Kingdom ' / 1.111, 1:1 les[p)
HIN
HINE, J and M CUNDILL (1 994). Economic assessment of road projects: do our current procedures tell us what we want to know? Paper presented to the International Workshop on Impact Evaluation and Analysis of Transportation Projects in Developing Countries, Bombay, 13-16 December 1.994. INTERNATIONAL WORKSHOP ON IMPACT EVALUATION AND ANALYSIS OF TRANSPORTATION PROJECTS IN DEVELOPING COUNTRIES Bombay, December 13-16, 1994 ECONOMIC ASSESSMENT OF ROAD PROJECTS: DO OUR CURRENT PROCEDURES TELL US WHAT WE WANT TO KNOW? By John Hine and Mike Cundill, Overseas Centre Transport Research Laboratory Overseas Centre, Transport Research Laboratory Crowthorne Berks RG 1 1 6AU UK Tel: 44 344 773131 Fax 44 344 770719 ECONOMIC ASSESSMIENT OF ROAD PROJECTS: DO OUR CURRENT PROCEDURES TELL US WHAT WE WANT TO KNOW? By John fine and Mike Cundill, Overseas Centre, Transport Research Laboratory No-one can question the importance of roads and road transport. Currently, developing countries spend around nine per cent of final consumption expenditure on transport and almost always road transport is by far the dominant mode. Expenditure on road investment and road maintenance is usually between 0. 15 and 1 per cent of GDP. Most of the goods we consume are provided by the market but this is not the case with roads; there is no competitive market in the provision of road space. For most goods, if there is an under-supply, prices and profits rise to encourage further production and conversely, if there is an over-supply, prices and profits fall to discourage production. In some cases, the feedback may be long-term rather than short-term but, in the course of time, profitability and output figures will show if an investment is a waste of resources. Unfortunately, there are no such feedback mechanisms with the provision and maintenance of road space. Governments are totally dependent upon the judgement of engineers, economists, planners and researchers to tell them when, where and how many roads should be built and to what standard they should be maintained. Nor can we rely on public opinion to strike the right balance. Here the point of view of the individual consumer takes precedence and no thought is usually given to the costs of provision; in the UK an empty motorway is a cause for driver celebration whilst the overloaded M25 is regarded as a 'disaster' to be blamed on the Government and planners alike. It is for these reasons that the proper economic assessment of road investment is so critically important. Appraisals and Evaluations The terms appraisal and evaluation are often used interchangeably. Here we shall try to use the term 'appraisal' to refer to the process of planning a future investment, whereas 'evaluation' refers to an investigation that happens after an investment is made. The main purpose of an appraisal is obvious, it is to help plan the best (most economic) solution to a particular problem. Evaluations are usually carried out to reassess the value of a project but one could question the degree to which the lessons learned are incorporated into future appraisal methods. 1 An appraisal usually compares a near-future forecast of an investment cost with the difference between two uncertain longer-term future outcomes, the 'do nothing' case and the investment case. The most basic road investment appraisal will involve assumptions and forecasts about future levels of road maintenance, road deterioration, vehicle operating costs (VOCs) and traffic. An evaluation is carried out with partial hindsight. It is carried out after the main investment has taken place and so compares a known investment with one unknown outcome which can never be tested (the 'do nothing' case) and some knowledge of the other outcome (the investment case). In practice, only a small proportion of the assumptions made in an appraisal concerning the project outcome are checked in the evaluation. Of all the things that can be measured, traffic flows are one of the most straightforward, yet they are seldom measured with the required degree of accuracy. Similarly, adequate vehicle operating costs are rarely collected. Where the evaluation takes place very soon after the investment is finished, very little extra is known about the outcome that was not known at the time of the appraisal, yet the forecast 'life' of the investment may be for twenty years. Short-term levels of diverted and generated traffic might be measured, though this is not easy to do, but the critical element of traffic growth cannot be verified nor can the future levels of road maintenance, road condition or agricultural output. Particular care needs to be taken when the results of many evaluations are summarised together -and we must not be misled into thinking that we have, even more or less, the 'right answer'. (See, for example, Pohl and Mihaljek analysis of 1015 World Bank projects which indicated that the economic rate of return at appraisal was 18 per cent and at project completion was 14 per cent.) Because of the range of uncertainties and assumptions associated with both appraisals and evaluations it is vitally important to subject them both to careful scrutiny. At appraisal, there is often pressure from the client (who is nearly always the Highway Authority or lending agency and rarely the Ministry of Finance) to show a high positive rate of return for the chosen investment. Inevitably transport planners are encouraged to adopt assumptions for the project which produce the required answer. It was partly for this reason that so much effort went into the production of the Transport Research Laboratory's Road Transport Investment Model (RTIM), the World Bank's Highway Design and Maintenance Standards Model (HDM) and the Road User Cost Study in India. The desire was to put road appraisal into a systematic framework and reduce the latitude that planners had before. Usually the best way of producing a high rate of return for the chosen project is to postulate a very poor 'do nothing' scenario and contrast this with the chosen investment case. In Africa, the case for gravelling a low traffic earth road is often made (via HDM or RTIM) on the basis of the very minimum level of road maintenance in the 'do nothing' case. The case for gravelling will usually fail if frequent grading is tested as an alternative. 2 Through experience we have found that, more often than not, the best option at appraisal is the 'minimum do something'. Unfortunately this is rarely explored. Often the evaluation provides information on only one investment alternative and the test of the 'minimum do something' still needs to addressed in evaluations as well as in appraisals. The Components of Uncertainty Given that assumptions have to be made in appraisals and evaluations about key elements, such as road deterioration, VOCs and traffic, what are the implications? If these elements are stable and predictable, then it may not matter. It is instructive, therefore, to consider what is known about their variability and uncertainty. Investment Costs This is commonly regarded as one of the most predictable elements of a road project. In 1982, an analysis of 32 road projects financed by the European Development Fund found that the mean final cost of projects was 16.5 per cent higher than the original estimated costs, with a standard deviation of 35 per cent. The minimum value was 32 per cent less than predicted, while the maximum was 100 per cent more than predicted. Road Deterioration Although this is an area which has been well researched, the predictability of the commonly used models appears poor. In attempting to explain the roughness of unpaved roads (roughness is the most critical factor in vehicle operating costs) Paige-Green (1989) summarises his own work and that of the HDM in the following:- 'In all of these models, as in the Brazilian models , the r-squared values are poor (only between twenty and thirty per cent of the variation is accounted for by the models). Low r- squared values for the prediction of pavement distress are generally reported in the Literature (Middleton and Mason, 1987). The HDM3 Manual identifies high prediction errors (95 percentile confidence intervals of 20 to 40 per cent) as being typical of this type of study (World Bank, 1985) and ascribed them to large variability of material properties, drainage, surface erosion and the high roughness levels of unpaved roads..' The modelling of the deterioration of paved roads is very complex and some studies have achieved much better explanation than the 20-30 per cent-r-squared values quoted above. However problems arise when we compare the results of different studies. For example through a reanalysis of the AASHO road test data and using different model formulations Shook and Finn are reported to have found a damaging power of four times that of Liddle for a 66.5 kN wheel load while Konder and Krizek have found a damaging power of 60 times that of Liddle for a 4.5kN wheel load (Lister 1981). 3 More recently TRL's own research on pavement deterioration within the tropics has shown that climatic factors often dominate the performance of paved roads and that the mechanisms of failure have been misunderstood. Traffic Research carried out by Howe (1972) in Kenya suggests that rural traffic flows are very variable and that errors are related to the duration and timing of the traffic counts. For a road carrying 100 vehicles per day, the estimate of traffic flow derived from a one-day count would have a 95 per cent confidence level of plus or minus 50 per cent. Increasing the duration of the count to a full week would increase accuracy and reduce the confidence interval to plus or minus 30 per cent. The errors associated with counting fall as the annual average traffic flow rises, but even with a flow of 1000 vehicles per day, a one-week count has a 95 per cent confidence interval of about plus or minus 16 per cent. More recent research carried out by the Transport Research Laboratory in a number of countries has shown that it is possible to get very large fluctuations in traffic flow. They may be caused by nearby road closures, feast days, holidays or even an end-of-month variation but often the causes are quite unexplained. Vehicle Operating Costs Large scale studies have been carried out which relate road condition to vehicle operating costs in Kenya, the Caribbean, Brazil and India. The results are incorporated in RTIM and HDM. Many of the component relationships used in the models appear to be extremely complicated (see for example the HD)M/Brazilian speed/fuel consumption relationship). At first one might think that such complexity in a model would indicate good overall accuracy but unfortunately this is not the case. Although, within each study, many of the central quoted relationships may seem plausible, the key relationships are often statistically poor. An important relationship used in the calculations is the effect that road roughness has on vehicle repair bills. The equations used to derive this from the field studies have r-squared values of only around 50 per cent. To make matters worse, there are major differences between the results of the different field studies suggesting that the scatter of the results under-estimates the possible inaccuracy. It is sometimes suggested that before VOC relationships are used they should be 'calibrated' to suit local conditions. Of course this is sensible, but the -term 'calibrate' is something of a misnomer. It implies some small adjustment, a fine tuning, but this is often very far from the case. For example a nine-year-old truck, travelling 112,000 km per year on a road of moderate roughness (4,500 mm/km on the BI scale) would consume 6.4 times the value of spare parts according to RTIM as it would according to the Indian VOC relationships. Since these major studies, at TRL, we have carried out a number of smaller studies. Unfortunately, each new study appears to produce a new set of relationships. 4 Agricultural Output For many rural road projects, appraisals and evaluations include some impact on agricultural production and the wider economy. Impact estimates in this area are usually the most controversial and contentious element of any assessment. There is no simple, widely accepted method with which to estimate the benefits from road investment. A number of model frameworks have been developed to assist with estimating benefits and perhaps the most well known of these is the 'Producer's Surplus Model' ( Carnemark et al. 1976). However the weakness of this model (and most others that attempt to address the same problem) is that the key element of agricultural response has to be estimated by the user. As far as we are aware, there are no calibrated, predictive models of -road investment and agricultural production in general use. As with most forms of economic development, agricultural production does not grow in a smooth and even manner. Usually there are a large yearly variations in output which mask any changes brought about by the road investment (see below). The Future There is old Danish proverb: " It is very difficult to make predictions, especially when they concern the future". Roads are often appraised over a period of twenty years. What confidence can we have in predicting traffic, road condition and changes in VOCs over that period? What type of vehicles will be running on Indian roads in fifteen to twenty years time? Particular Problems with Post Evaluations In our experience, often the more information that is collected during an evaluation study the more complex the situation appears to be. Evaluations which are carried out soon after the investment has been made can rely, unchallenged, on previous trends and the standard models used in the appraisals. The difficulties start when attempts are made to reconcile the results of 'repeat' and 'control' surveys with what we believe should have happened based on our preconceived notions, trends and models. We have seen examples when traffic has apparently fallen after a road has been built and on other occasions, agricultural output has fallen in the vicinity of both the control road and the project road. We have also seen examples where dramatic increases in traffic have occurred which appear to be little related to the road investment. And sometimes there seems to be no apparent effect. 5 Too often, the changes we see in traffic and agricultural output appear to be a function of changes in the national and local economy, the weather and of factors other than the road investment. Traffic naturally responds td~changes in economic activity. Large year-by-year fluctuations in both the composition and the total output of agricultural production can be caused by the weather and by farmers responding to changes in market prices. Major fluctuations in coffee and tea prices in Kenya have made it particularly difficult to interpret road evaluations in that country. An example of the problems of using controls is shown in a series of road impact studies carried out in Thailand under the direction of J H Jones (1973). Through the use of control reference areas, the studies showed how increases in agricultural production could be attributed to the opening of new roads. However, since the completion of the studies, Jones has suggested that variation in the level of internal security could also be a possible explanation for the results. Three of the case studies had reference areas close to Thailand's (military unstable) borders. Another problem arises with traffic, even if we are absolutely confident that we know what the level is following an investment. The question is -what would have been the traffic level without the investment? Strangely enough, if we believe that the road investment has induced a substantial amount of traffic then the benefits arising from the investment is less than it would have been had we believed that the traffic levels would have been almost the same without the road. This is because 'generated' traffic benefits or the benefits arising from traffic 'induced' by the investment are valued less than benefits from 'normal' traffic which would have occurred anyway. Evaluations of the engineering performance of a road are also far from straightforward. Usually difficulties arise with trying to find out: a) exactly to what standards the road was constructed in the first place b) exactly what maintenance has been carried out since construction c) exactly what maintenance will be carried out in the future Time after time we have discovered that roads have failed not because of an inadequate design but because the road was not constructed to the specified design in the first instance or because inadequate maintenance was carried out. Wider Issues There have been a number of studies which suggest that maybe the conventional transport cost appraisal is too limited to provide an indication of the total impact of road investment. Road investment could promote development through widening markets, encouraging competition, improving information flows and provide multiplier effects. Various cross- sectional studies have been carried out which suggest that there may be a relationship between development and transport and infrastructure. 6 Through an analysis of inter-country data, Antle (1983) has found a relationship between agricultural output and gross domestic product spent on transport and communication industries per unit of land area. Similarly, a study in Bangladesh by Ahmed and Hossain (1990) has found a relationship between infrastructure and accessibility, and various measures of development. In contrast, a cross-sectional study of the Ashanti Region of Ghana carried out by the Transport and Road Research Laboratory found there was no positive relationship between roads and development (Hine et al 1983a, 1983b). If we do find a relationship, how does this help us? We need to know whether the relationship is causal or not. In the first instance, it can be argued that expenditure on transport and communications per unit area are proxies for the density of demand, which in turn may be a more important influence on the value of agricultural production than transport expenditure itself. High value agricultural products can only be afforded by a wealthy population. In the second instance, which comes first, road investment or development? Clearly a minimum level of accessibility is required before development can take place but what happens once that basic minimum has been achieved? Do roads promote development or does development encourage roads? Road investment does not happen in an economic or political vacuum. Conventional appraisal techniques assume that road investment follows demand. Furthermore the dynamic and resourceful village leaders (in the more developed villages?) are more likely to get public investment for their villages than their more indolent colleagues. Thirdly, if we believe road investment has unforseen development 'multiplier' effects, who is to say that we could not have achieved the same result through other forms of public expenditure, in irrigation or housing, for example. If we address these issues within an evaluation, we always need to ask ourselves 'What development would have happened without the road investment?'. It is often extremely difficult to find a satisfactory answer, even if we have controls. If we accept that road investment, in general, does promote development in ways which are not easily measured by the conventional transport appraisal how does this help us plan road investment? We still need to make choices about particular schemes and it is very difficult to apply knowledge about a global relationship to the choice between a gravel or a paved road, or the choice between a two-lane road and a motorway. Where do we go from here ? It is very clear that we live in an extremely complex environment and that we cannot model the whole of this complexity. But we do need procedures to help us with road planning. There are no automatic market feedback mechanisms to help us. We also know that it is relatively easy for planners to manipulate complex procedures to ensure that a road investment is made to look worthwhile. 7 What can we do? The argument comes back to choice. We can have little confidence in our abilities to compare the economic rates of return of a road project with an irrigation project. The limited economic accuracy and the widely different patterns of externalities would render such comparisons as dubious. However we are on much firmer ground when we compare different road projects using the same procedure, even if the model chosen is rather poor at representing all the complexity of reality. Perhaps we should place much more emphasis on asking planners and economist to rank alternative road proposals and less on trying to judge the economic merits of projects in isolation. In order to help improve choice, we need relatively simple and easy to understand models which focus on the main characteristics of road projects and help us to distinguish between them. Simple procedures and models which help us to measure and model traffic, roughness, vehicle operating costs and road deterioration will help. If we wish to use a relationship between road investment and agricultural production, we need to concentrate on the forecast change in transport costs of the different road investments and the elasticity of agricultural supply. With this information we might have some basis for predicting the likely response (see Hine 1982). In the short term, it is important to try, test and ground-base all the models we use. We should consider cross-sectional and time series studies. Cross-checks must be applied where possible, for example by comparing freight tariffs with estimated VOCs. In the longer term we believe that there will be a move towards developing simpler (and possibly more robust) modelling procedures. Acknowledgements The work described in this. paper forms part of a programme of research of the Overseas Centre, of the Transport Research Laboratory (Programme Director: Dr. J Rolt) and was funded by the Engineering Division of the Overseas Development Administration, UK. References Ahmed R and M Hossain. (1990) Developmental impact of rural infrastructure in Bangladesh. IFPRI Research Report 83. Washington DC. International Food Policy Research Institute. Antle JM (1983) Infrastructure and aggregate agricultural productivity: international evidence. Economic and Cultural Change 31: pp 609-619. Hine JL, Riverson JDN and E A Kwakye. (1983a) Accessibility and agricultural development in the Ashanti Region of Ghana. Department of the Environment, Department of Transport TRRL Report SR 791, Crowthorne. Transport and Road Research Laboratory. 8 Hine JL, Riverson JDN and E A Kwakye. (1983b) Accessibility, transport costs and food marketing in the Ashanti Region of Ghana. Department of the Environment, Department of Transport TRRL Report SR 809, Crowthorne. Transport and Road Research Laboratory. Hine JL (1982) Road planning for rural development in developing countries: a review of current practice. Department of the Environment, Department of Transport TRRL Report LR 1046, Crowthomne. Transport and Road Research Laboratory. Howe JDGF (1972) A review of rural traffic-counting methods in developing countries. Department of the Environment RRL Report LR 427, Crowthome. Road Research Laboratory. Lister NW (1981) Heavy wheel loads and road pavements damage relationships. Symposium on heavy freight vehicles and their effects. Organization of Economic Cooperation and Development, Paris. Middleton, DR and Mason JM (1987) The use of site-specific truck traffic to evaluate the performance of surface-treated pavements. Transportation Research Record, 1106, Transportation Research Board, Washington DC. ppl123-132 Paige-Green P (1989) The influence of Geotechnical Properties on the Performance of Gravel Wearing Courses. Phd Thesis, University of Pretoria. Pohil G and D Mihaljek (1992) Project evaluation and uncertainty in practice: A statistical analysis of rate-of-return divergencies of 1015 World Bank projects. The World Bank Economic Review, Vol 6, no 2, pp 255-277. Jones JH (1973) Evaluation of the effects of highways: the Thailand case. Conference on Road Engineering in Asia and Australia, Kuala Lumpur. World Bank (1985) Highway Design and Maintenance Standards Study. World Bank, Washington DC. Crown Copyright 1994. The views expressed in this paper are not necessarily those of the Department of Transport or the Overseas Development Administration. 9 A Study of 33 villages in the Ashanti Region of Ghana (Main cash crops: qooa maiz Accessible villa~ges - more dependent o n non-farming jobs - more trip making - more success in loan finance - greater proportion of cassava sold Inaccessible villages - more dependent on agriculture - more labour input into farming - more sheep, goats and poultry - .mo~re.,cocoa grown and sold. per farmer -greater use of fertiliser, insecticide, .extenso and tractor hire -greater proportion of plantain sold No difference - cocoa sales per hectare - maize yields. or proportion of maize sold Potential Improvement in Farmw @R* Prices a) after upgrading earth to gravel surface road length Increase. In farm-gate prices(% Maize Yam Plantain 5 km 0.08 0.11 0.09 20 km 0.29 0.30, 0.24 50 km 0.67 0.50 '0.37 b) after conversion of footpath to earth road length of footpath to be changed to vehicle access 2 km 5 km 20 km Increase In farm- gate maize prices () 4.3 11A4 70.6 / 0~ U H CN .. rJ2 0 UC. C
1 0 ~ ct i
-4 .iii m i E-( 11 1 1 i 0 U. """. pit .Ii ct . 0 .c. ) 1-.
I_.-(:. 1:M
: i i 11i i1 ........ 1 FEEDER ROAD PLANNING Vehicle access is the key! - The quality of the road surface is unimportant. If upgrading existing vehicle access conventional transport cost savings will be small and development benefits will be very small or non-existent. If providing new access to an area transport cost savings can be large and development benefits may be substantial. It is best to invest in bridging, minor drainage work and other small scale measures to ensure that all villag es have direct vehicle access and routes are kept open.