Cost forecasting for public construction projects

The administration of public construction projects comprises stages that go from the elaboration of the engineering project up to the conclusion and delivery of the work. This includes budgeting, bidding, contracting, and execution phases. However, it is in the budgeting phase that it is possible to determine, through cost forecasting, whether a particular undertaking is feasible (Paula and Garcia, 2012).

Cost has been an important variable for contracting companies to carry out public works through a bidding process. However, ^{Tisaka (2006)} points out that choosing a company based only on the lowest price is not the best criterion when it comes to achieving the objectives of said process. This choice, based on the lowest cost criterion, levels the quality of the construction downward. In the context of public construction works, there are several examples of bids whose work did not start, was poorly finished, or even dropped altogether (^{Borba and Marinho, 2019}; ^{Cunha and Caffé Filho, 2019}; ^{Jorge and Ribeiro, 2006}; ^{Oliveira, 2016}; ^{Tisaka, 2006}). When on hold, these projects can bring several problems for public administration, which implies losses for taxpayers.

A possible flaw in the management that may cause uncontrolled costs directly influences the final results of the project (^{Silva, Corrêa, and Ruas, 2018}). The public sector, for example, has difficulty completing works such as the construction of kindergartens, schools, hospitals, sport courts, basic sanitation systems, roads, ports, and airports (^{TCU, 2018}).

In 2006, upon realizing the difficulties public authorities face to conclude the execution of construction projects, the Federal Court of Auditors (TCU) carried out a survey to diagnose the situation of unfinished public works. The audited works were mostly linked to the executive branch.

A set of 400 unfinished construction projects was presented in the TCU report (2018). The total cost attributable to these projects amounted to R$ 3,5 billion. Of the 400 works, 130 were carried out directly by the federal government and 270 by states and municipalities, but all used federal resources. Figure 1 shows the causes of stoppage hitherto identified in the report by the Court of Auditors.

^{Source: TCU (2018)}

Figure 1 Causes of work stoppage. 

In Figure 1, it can be seen that the main cause for stoppage of the works analyzed by the TCU's Court of Auditors was budgetary problems. As Paula and Garcia (2012) state, the ability to predict construction costs can result in more accurate proposals.

Given that budget is an important factor in the stoppage of public construction projects, it is expected that mechanisms to help decision makers to forecast real costs before starting a work may be developed in relation to financial issues, so as to diminish the probability of interruptions. These premises endorse the importance of developing a computational model capable of forecasting the real cost of public undertakings.

Artificial Neural Networks - ANN

Artificial neural networks are computational models inspired by the functioning of the nervous system, namely the functioning of neurons (^{Silva, Spatti, and Flauzino, 2016}). Neurons are cells capable of perceiving changes in the environment, maintaining communication with each other through chemical and/or electrical processes, as well as commanding bodily responses (^{Bear, Connors, and Paradiso, 2002}).

Communication between neurons and information processing essentially depends on three integral parts: dendrites, cell body (or soma), and axon. Similarly, the functioning of an artificial neuron occurs involving the following integrating parts: the synaptic weights, sum, and activation functions. The relationship between these parts can be seen in Figure 2.

Source: ^{Silva et al. (2016)}

Figure 2 Artificial neuron.  

The mathematical operations thatcharacterizethefunctioning of the neuron begin with the multiplication of each input X _j (connected to the neuron k in j) and multiplied by the synaptic weight W _kj ). After the multiplications, an adder is activated, transforming the set of multiplications into a linear combination Σ. The result of the linear combination u_k, plus a bias b _k , is applied to an activation function g( ) (^{Silva et al, 2016}).

Mathematically, the functioning ofan artificial neuron is given in Equations (1) and (2).

^{Haykin (2001)} defines the architecture of an ANN as the way in which neurons are arranged together or structured in layers. The existence of at least one layer between the inputs and the output layer of an ANN is what characterizes a multilayer network. Among these multilayer networks, the Multilayer Perceptron (MLP) type stands out (Figure 3). Multilayer Perceptron networks are applicable to several types of problems, such as approximation of functions, forecasting, and optimization.

Source: ^{Haykin (2001)}

Figure 3 Multilayer Perceptron (MLP).  

The use of an MLP network for a given project requires the designer to define the specifications (topology) of the network beforehand. This includes, for example, the number of input signals, the number of hidden layers, the number of neurons in each hidden layer, and the number of neurons in the output layer. There are no pre-established rules for choosing the network topology. This choice depends on the problem in question and is usually chosen through computational tests (^{Rocha, 2018}).

The training of an MLP-type network is carried out through supervised learning and the error backpropagation algorithm. This algorithm is based on the error correction learning rule and consists of two phases: propagation (forward phase) and backpropagation (backward phase) (^{Haykin, 2001}).

In the propagation phase, the input signals are multiplied by the synaptic weights, applied to the adder to form a linear combination, and the result is applied to an activation function. This process is carried out on all neurons and on all layers until reaching the output layer (^{Silva et al., 2016}). When the process reaches the output layer, the result is examined to check for the existing error between the network output and the desired signal (^{Haykin, 2001}; ^{Rocha, 2018}; ^{Batista, 2012}).

The error obtained in the propagation phase is used to update the weights of the MLP layers, so that this update is made starting at the last layer until it reaches the first one. This process of updating of weights is known as backpropagation (^{Haykin, 2001}). Figure 4 shows the operation of the backpropagation algorithm in an MLP network.

Source: Adapted from ^{Asteris et al. (2016)}

Figure 4 Backpropagation algorithm.  

In short, the successive application of the forward and backward phases implies the minimization of error and consequent convergence of the network (^{Silva et al., 2016}). Optimization methods are applied for derivation of the iterative expressions for updating weights.

Database

The data used in the modeling process were obtained from an online platform of the MEC (^{MEC, 2019}). This platform is called the Integrated Monitoring, Execution and Control System (SIMEC). Authorization from MEC was requested to access the data because these are not public domain data. A complete description of the methodological acquisition procedure and an analysis of this database can be found in the work of ^{Alvarenga (2019)}. The data used in the present work were provided by this researcher.

In order to avoid a high number of renovation and adaptation projects in the sample, the following filters were applied in the listing of projects: projects that were 100% completed, projects with costs above R$ 1 000 000,00, and projects in federal universities, federal institutes, and university hospitals. In the first filtering, a total of 2178 contracts were considered for the composition of the database, broken down into 109 universities and university hospitals in 27 federation units located in 460 municipalities.

The sample composition was quite diverse and included information on constructions carried out in several Brazilian municipalities in all regions of the country. The completion of the works took place in the period between December, 2006, and August, 2017, and the report was extracted from the system in early September of 2017. Out of the total data, 11% refer to works carried out in the North of Brazil, 22% in the South, 27% in the Northeast, 27% in the Southeast, and 13% in the Midwest. (^{Alvarenga, 2019}).

A second filtering was applied in order to find only samples that contained information on both dependent and independent variables to be used in the modeling process. After performing this second filtering, the database was reduced to 1094 samples, which were then used for modeling.

Modeling

The modeling process sought to be in line with the concepts of experimentation, abstraction, resolution, validation, and modification presented by ^{Bassanezi and Ferreira (1988)}. The modeling phase was therefore carried out in three stages: choice of the type of ANN and construction of the algorithm, specifications and training of the ANNs, and testing of the networks.

Choice of type of ANN and construction of the algorithm

The choice of the type of ANN usually depends on the problem to be modeled. MLP networks are commonly used for forecasting problems. Thus, a MLP network was chosen for cost forecasting in the context of this study. It was also decided that an algorithm would be written to carry out the MLP training and testing phases.

It is worth noting that the choice of the number of separate samples for training and testing does not come from statistical methodologies, but from tests related to the learning capacity of the ANN, that is, it depends on the problem to be solved. In some problems, this separation can occur according to the ratio of 70/30 (70% for training and 30% for testing), or even 90/10 (90% for training and 10% for testing).

The topological characteristics of the network regarding the number of layers, learning rate, stopping criteria, number of neurons in each layer, and bias were defined based on various training runs and the validation of the network.

It is also worth mentioning that there are no mathematical mechanisms for defining the characteristics of the network, thus giving an empirical character to this choice. After conducting several training sessions with different topologies, the best results were chosen, and the respective topologies were presented.

Before carrying out the training of an ANN, it is always advisable to normalize the data, so as to limit them to the interval [0,1] (^{Doğan, Arslan, and Ceylan, 2015}). Oneway to achieve this normalization is to use Equation (3).

where A _n is the normalized sample, A¡ is the original sample, and min(A) and max(A). These are the lowest and highest value among all samples of the variable to be normalized, respectively.

The specifications used in the ANN for cost forecasting were: four inputs, one hidden layer, eight neurons in the hidden layer, and one output (cost). Figure 5 illustrates the ANN's topology and training process in cost prediction.

Source: Authors

Figure 5 Topology of the network used to predict public work costs.  

The value predicted by the network in each loop is compared with the measured value. Therefore, it was defined as a criterion for stopping the network when the difference between the error of one looping and the other one is not less than 10^-7. Otherwise, the network continues to update the synaptic weights.

ANN testing

The testing phase of the ANN for cost forecasting consists of applying the forward phase of the backpropagation algorithm only once in new data. This phase differs from the training phase in that only one looping is performed without updating weight; the weights used are those obtained during training, and the calculated error is analyzed for model validation purposes. Figure 6 shows how the ANN testing process for cost forecasting takes place.

Source: Authors

Figure 6 Network topology used to validate the cost prediction model. 

Model Evaluation and statistical analysis

Before performing the evaluations of the results, the sample values were denormalized according to Equation (4).

For the results of the presented models, we analyzed the statistical characteristics of both the training and test phases. These analyses consisted of evaluations using accuracy and correlation measurements, as well as hypothesis tests.

With the values calculated by the neural networks, the mean absolute percentage error (MAPE) between the outputs of the network and the real values were calculated. MAPE is mathematically defined according to Equation (5) (^{Belalia et al., 2017}; ^{Maues, 2017}).

where X _i is the measured or observed value (real value); Y _i is the respective approximation to X_i, obtained through the mathematical model; and n is the number of samples evaluated. The MAPE was used to verify the accuracy (or assertiveness)of the model, and it was determined from the percentage of predicted values belongingtothe interval [L_s,L_i] defined by Equations (6) and (7) (^{Maues, 2017}).

In these Equations, L _s and L _i are the upper and lower limits of the assertiveness range, respectively, and V _R is the real value of the database.

A correlation analysis was also carried out to verify if there was a linearity relationship between the predicted and real values. In addition to the accuracy and correlation estimates, comparisons were made between the box plots of the actual values and the values obtained by the models in order to graphically verify differences between the analyzed samples.

As graphic analyses alone are not enough to infer the existence or not of differences between the sets of the analyzed samples, it was necessary to carry out hypothesis tests. First, the Kolmogorov-Smirnov test was performed to infer the type of data distribution. The test revealed that samples did not come from a normal distribution. Then, the Mann Whitney test was carried out to check for significant differences between the real values and those obtained by the neural models (^{Bussab and Morettin, 2003}).

Training of the ANN for cost forecasting

The input variables used in the training phase were the area of the facility to be constructed, the number of inspections, the estimated budgeted cost, and the contracted cost. The topology of the network used is described in Table 1.

Table 1 ANN Topology 

Source: Authors

994 of the total of 1094 samples were randomly selected and used to carry out the training of the neural network. A MAPE equal to 9,62% was obtained. From the MAPE value, an interval was established to calculate the accuracy. Of the 994 values referring to the total cost of the works used in the training of the ANN, 878 were within the established interval, that is, the model obtained an accuracy of 88,33% in the training phase. An overview of the value sets within and outside the established interval is presented in Figure 7.

Source: Authors

Figure 7 Results of the training phase of the ANN for cost forecasting. 

A subset of samples is presented in in Figure 8 to better understand the relationship between the estimated values and the error interval established for the analysis of the model. However, in the presented samples, it appears that the estimated values that do not belong to the error interval are visibly close to the upper or lower limit.

Source: Authors

Figure 8 Results of the ANN training phase for cost forecasting (some samples). 

The main objective of the training phase of an MLP-type ANN is to find the weight matrices that minimize the error between the real values and the network's output. If the weight matrices are known, the neural model can be applied to new data, and the results can then be extrapolated. With that in mind, the following are the synaptic weight matrices obtained in the training process for cost forecasting:

and

where w ₁ represents the weights of the first layer and w ₂ , the second layer. It is important to note that the number of lines of w ₁ corresponds to the number of neurons in the hidden layer, whereas the number of columns is equal to the number of inputs plus 1 (bias). The dimensions of w ₂ refer to the number of output variables (row) and the number of neurons in the hidden layer plus 1 (columns).

Testing of the ANN for cost forecasting

The test of the neural model was performed usingthe matrices presented above. 100 samples were separated and used for this phase. It is worth mentioning that the samples for both training and testing were randomly chosen.

The MAPE value found in the testing phase was 9,14% and thus, in an analogous way to what was done in the training, an interval was established to check the accuracy of the model. Ninety samples were within this interval, that is, the accuracy in the testing phase was 90%. In Figure 9, it is possible to see which estimated values were within and which were outside the interval.

Source: Authors

Figure 9 Results of the ANN test phase for cost forecasting.  

An interesting observation to make when analyzing Figure 9 is that there are almost no values with great differences concerning the error limits. This suggests that the trained model has a strong generalization 'power'.

Statistical analysis of cost forecasting

In this subsection, a statistical analysis is presented which considers the calculation of Pearson's correlation coefficient, the characteristics of the distributions of the real and the network output values, and the hypothesis tests.

The correlation between the network output values and the measured values was 0,9921 in the training phase and 0,9912 in the testing phase. The correlation analysis is important, as it demonstrates the extent to which two variables are linearly related. Figure 10 shows the relationship between the analyzed variables and the line that best approximates them for the training and the testing phases.

Source: Authors

Figure 10 Correlation between the measured values and the ANN output: A) Training phase, B) Test Phase. 

The high correlation obtained both in the training and test phases of the network means that there is a linear relationship between the real values and those obtained by the network. This fact corroborates the results related to the small number of values outside the error range.

The fact that there is a linear relationship between the real and the network output data shows that, in this particular case, in addition to small percentage error, there is a relationship of proportionality between variables. This trend indicates that, when a real value increases (or decreases), the network result also increases (or decreases). This analysis is important, as it can encourage further research on models complementary to ANN (to create a hybrid model). That is to say, the ANN output data can be used as an input to another model and generate enhanced results. As for the 'behavior' in terms of distribution, the box plots in Figure 11 show how the data in the training and testing phases are distributed around the median.

Source: Authors

Figure 11 Distribution of measured values and ANN output for cost forecasting: A) Training phase, B) Test Phase. 

The distributions shown in Figure 11 allow inferring and assuming that there is not 'much difference' between the sets. However, measurements and the application of techniques are needed to verify whether or not there are significant differences between values coming from the ANN and the real values. To achieve this goal, hypothesis tests were performed. First, the Kolmogorov-Smirnov normality test was carried out using the network output values and measured values in the two phases (training and testing) of the construction of the proposed model.

Results of the tests in all data sets pointed to the rejection of the null hypothesis (normal data), with a significance level of 5%. As the null hypothesis was rejected, that is, the data did not come from a normal distribution, the Mann-Whitney U test was applied to verify the existence of significant differences.

The Mann-Whitney U test was applied to the results obtained in the training and in testing phases for the case of cost forecasting. In the training phase, the test showed that the hypothesis that the network output and real values are samples of distributions with equal medians should be rejected at the significance level of 5% (p-value = 0,0307). In the testing phase, the results pointed to the non-rejection of this hypothesis (p-value = 0,555).

Estimating the total cost of the construction based on the forecast of cost overruns

During the development of this study, we noticed possibility of obtaining a system capable of estimating the final cost of the construction projects based on the forecast of overruns. This system consists of training an ANN whose desired output is the cost overrun. After the neural model provided such value, it was added to the contracted value, thus leading to the total cost of the work. This procedure is summarized in Figure 12.

Source: Authors

Figure 12 Total cost forecast through forecast of cost overruns.  

Table 2 shows the values of the main evaluative metrics regarding the total cost prediction of the work from the additive values provided by the ANN.

Table 2 Results obtained for training data 

Source: Authors

Results obtained in the testing phase of the total cost forecast of the work based on cost overrun values are shown in Table 3.

Table 3 Results obtained for testing data 

Source: Authors

In view of the results presented in Tables 2 and 3, it is evident that the methodology based on the use of the system proposed in Figure 12 presents relatively better results than those obtained with the use of the model directly trained for cost forecasting.

These error percentages of cost forecasting are promising and demonstrate that this research brings an important contribution to the proposal of collaborations aimed at improving the planning process of construction projects, given that, according to ^{Hyari et al. (2016)}, conceptual cost estimates relying on traditional methods are inherently associated with high percentage errors, usually in the range between 20,8 and 27,9%.

For comparison purposes, one can observe the results obtained by ^{Hyari et al. (2016)}, who carried out a similar work, applying a neural model to forecast costs. They obtained 28,2% of MAPE (testing phase). However, it is important to note an important difference between their work and ours, namely the amount of values available for training the network, as it is known that the performance of ANNs increases with the amount of training samples. That is, as the number of samples used in the present study is much greater than those used by ^{Hyari et al. (2016)}, better results were expected.

As the weight matrices were made available in the text, the forecasting model can be implemented in more popular software such as Microsoft Excel, and thus be used by engineering professionals (even without any knowledge of ANNs) in their professional practice, provided they have access to the network. Furthermore, this model can be 'embedded' in electronic devices such as computers and cellphones.