1.2.1. Gradient descent algorithms

Error backpropagation (BP): It is called backpropagation algorithm because the error is propagated back to update the weights and the biases using an optimization technique (gradient descent), which minimizes the error (Dixit and Londhe, 2016). The backpropagation propagates the input forward through the network and the sensitivities backward, it updates the network weights and biases in the direction in which the performance function decreases most rapidly, which is the negative of the gradient. Its main drawbacks is that it is a local algorithm, which means that it can reach a local optimum instead of the global optimum, and that it is slow in converging (Demuth et al., 1992; Hagan et al., 1996).

It also exists the variant gradient descent with momentum (GDM), which objective is to faster the convergence of the algorithm. The momentum smooths the oscillations of the algorithm and allows the network to respond not only to the local gradient, but also to recent trends in the error surface, avoiding getting stuck in a local optima (Demuth et al., 1992; Hagan et al., 1996).

One of the main problems of the conventional backpropagation algorithm is that it is too slow. Therefore, there are other algorithms known as “faster training” that avoids this problems (Demuth et al., 1992):

Resilient backpropagation (RBP): ANNs usually use sigmoid functions in the hidden layers. These functions transform values from an infinite range to a range of 0–1 and of −1 to 1. This can cause the gradient to have a very small magnitude, which can cause small changes in the weights and biases, generating problems when using the gradient descent technique. The objective of this algorithm is to avoid these effects by determining the weights and bias by a separate update value (Demuth et al., 1992).

1.2.2. Conjugate gradient (CG) algorithms

Conjugate gradient descent algorithms are also classified as “faster training”. Common gradient descent updates the network weights and biases in the direction in which the performance function decreases most rapidly, which is the negative of the gradient. However, this does not necessarily produce the fastest convergence. Conjugate gradient algorithms also search along conjugate directions. There are different conjugate gradient functions (Demuth et al., 1992; Hagan et al., 1996):

• -

  Fletcher-Reeves (CGF): It computes the previous search direction by multiplying it by a factor that is the ratio of the norm squared of the current gradient to the norm squared of the previous gradient. It guarantees to locate the minimum of any quadratic function of n arguments in at most niterations (Demuth et al., 1992; Fletcher and Reeves, 1964).

• -

  Polak-Ribiére (CGP): In this case the factor is the inner product of the previous change in the gradient with the current gradient divided by the norm squared of the previous gradient (Demuth et al., 1992; Fletcher and Reeves, 1964).

• -

  Powell-Beale Restarts (CGB): In common conjugate gradient descent algorithms search direction will be reset to the negative direction of the gradient when the number of iterations is equal to the number of network parameters. The CGB technique restarts the direction of searching to the negative of the gradient if there is very little orthogonality left between the current gradient and the previous gradient (Demuth et al., 1992; Powell, 1977).

• -

  Scaled conjugate gradient descent (SCG): All previous functions use line search, which is computationally expensive. Instead of the line search approach, the SCG uses the model-trust region approach (used in the Levenberg-Marquardt algorithm) to scale the step size and combines it with the conjugate gradient approach (Møller and Moller, 1993; Demuth et al., 1992).

1.2.3. Quasi-Newton's algorithms

Newton's algorithms usually converge faster than gradient methods, but they are much more computationally expensive because they have to compute the Hessian matrix. Quasi-Newton's algorithms are based on Newton's method, but do not require calculation of second derivatives. There are two main algorithms (Demuth et al., 1992):

• -

  Broyden, Fletcher, GolFarb and Shanno (BFGS): It is based on conjugate directions combined with line search. It requires more computation in each iteration and more storage than the conjugate gradient methods, although it generally converges in fewer iterations. It is especially appropriate for small networks (Møller and Moller, 1993; Demuth et al., 1992).

• -

  One step secant algorithm (OSS): It deals with the problem of the storage needed for the BFGS algorithm. This algorithm assumes that at each iteration, the previous Hessian was the identity matrix (Demuth et al., 1992).

1.2.4. Levenberg-Marquardt algorithm (LM)

It is a variation of Newton's method that was designed for minimizing functions that are sums of squares of other nonlinear functions. When the performance function has the form of a sum of squares the Hessian matrix can be approximated as the product of the Jacobian matrix, that is much less complex than computing the Hessian matrix (Demuth et al., 1992; Hagan et al., 1996).

ANNs can perform very well in the training data set, but fail to perform in the test data set, which means that they do not perform well when adding new data. When the ANN is not able to generalize the problem is known as “overfitting”. This problem can be solved using regularization or early stopping (Standford University Machine Learning Course; Hagan et al., 1996). Regularization is a technique in which the different features are penalized by multiplying it by a factor λ, smoothing the function and making the network less complex (Standford University Machine Learning Course). One of the most common regularization techniques used in ocean engineering is the Bayesian regularization (BR). Mackay (1992) was the first one in applying the Bayesian framework to ANN modelling.

2.1. Prediction of wave heights

One of the main applications of ANNs is to predict wave heights, prediction of waves is an uncertain and random process that makes it ideally suited to NN modelling (Deo et al., 2001).

Deo and Naidu were the first authors that proved the advantages of ANNs over traditional methods for wave-height forecasting in 1998 (Deo and Sridhar Naidu, 1998). In 2001 Deo et al. predicted significant wave heights and average wave periods from the input of wind speeds (Deo et al., 2001). In 2002 Agrawal and Deo did it from current and previous wave heights (Agrawal and Deo, 2002). They found that when shorter prediction intervals are concerned, the NN had distinct superiority over the stochastic approach. However, this was not the case when a longer interval of prediction was studied. One year later, Deo and Jagdale showed that ANNs could also be used to predict breaking wave heights (Deo and Jagdale, 2003). In 2004 Agrawal and Deo concluded that ANNs outperformed more the statistical regressions when the input–output structure to be established is hazier and more unclear (Agrawal and Deo, 2004).

Also in 2004 Makarynskyy studied how to improve wave predictions with ANNs (Makarynskyy, 2004). A separate NN was developed to forecast each wave parameter at every location. One year later Makarynskyy et al. did a similar study with two approaches (Makarynskyy et al., 2005). They first used one ANN for each parameter (wave height and period) and in the second approach they used a unique ANN to predict both. They proved that the simplification of the NN sometimes gave better results. In 2005 Karla et al. conducted another two research. In the first one, they used satellite data and saw that this data needed pre-processing before applying it to ANNs (Kalra et al., 2005a). In the second one they also translated offshore satellite wave data and proved that the RBFNN performed better than the FFNN, but for higher waves the FFNN was more accurate (Kalra et al., 2005b).

In 2008 Jain and Deo conducted an experiment on AI tools to forecast waves in real time (Jain and Deo, 2008). They applied ANNs, genetic programming and model trees and saw that the differences between these 3 techniques were negligible. This same year Malekmohamadi et al. compared the performance of numerical wave modelling (NWM), ANNs and an hybrid model (Malekmohamadi et al., 2008). They concluded that although the NWM leaded to better results, such models were computationally expensive and needed long-time execution. Using artificial patterns for training ANNs leaded to more acceptable results, than those obtained using observed patterns. Also in 2008 Günaydin compared seven different ANN models with traditional regression methods (Gunaydin, 2008). They showed that the best performance was given when all the meteorological factors were included as inputs. Between all the meteorological inputs, wind speed was the most effective. The same year Mahjoobi et al. applied ANNs, Fuzzy Inference Systems (FIS) and ANFIS (Mahjoobi et al., 2008). They implemented a separate network for each parameter, and after one network for the three output variables. Results indicated negligible differences between the two approaches.

In 2009 Tsai and Tsai used ANNs to convert the signal of pressure transducersinto significant wave height, wave period, maximum wave height, and spectral peakedness (Tsai and Tsai, 2009). They compared the results with the ones obtained applying a linear wave pressure transfer function and found that for water depths greater than 20 m the NNs worked significantly better.

In 2015 other authors, Gopinath and Dwarakish proved that NARX models needed less data than conventional NNs to obtain the same results (Gopinath and Dwarakish, 2015). In 2016 Londhe et al. used ANNs to predict the errors in wave forecasting and added or subtracted these errors from the valued predicted by the numerical model, improving their predictions (Londhe et al., 2016). Salah et al. discovered that the hybrid model that combined SWAN and ANNs models outperformed both approaches, traditional and soft-computing (Salah et al., 2016). In 2017 Abhigna et al. saw that to predict extreme values of wave heights (both, high and low) ANNs performed worse (Abhigna et al., 2017). This same year Mudronja et al. stated that ANNs outperformed more traditional models when the number of inputs involved was high (Mudronja et al., 2017) and Sadeghifar et al. proved that the use of NARX networks improved the prediction of wave heights (Sadeghifar et al., 2017).

In 2020 Vieira et al. used ANNs to fill the gaps in wave records (Vieira et al., 2020). The inputs were wave and wind parameters. They performed several FFNNs with 3 (wave parameters) and 5 inputs (wave and wind parameters) and concluded that wind was not and essential parameter in that case, contrary to other studies. This same year, Stavarache et al. studied how to optimize ANNs for wave prediction and proposed to substitute the traditional trial and error approach by genetic algorithms (Stavarache et al., 2020). In 2021 Gao et al. concluded that LSTM networks had several advantages with respect to other soft-computing and conventional models (Gao et al., 2021). They also highlighted that a higher number of hidden neurons did not necessarily imply better results and that this number had to be adjusted to each wave height. This results were in line with Jörges et al. (Joerges et al., 2021).

In 2022 Sadeghifar et al. conducted an study in which they compared the performance of different soft-computing techniques (ANNs, ANFIS, Random Forest and M5P models) and proved that the M5P outperformed the others (Sadeghifar et al., 2022).

From all the type of networks, the FFNN was clearly the most used. Although in the last years LSTM neural networks are giving promising results (Fig. 1).

1. Download : Download high-res image (53KB)
2. Download : Download full-size image

Fig. 1. Frequency of networks used in wave height ANNs studies.

Referring to the architecture of the studies, all of them excepted one opted for a 3-layer neural network because adding more layers did not result in a significant improvement. In ocean engineering, shallow ANNs with one hidden layer are enough to forecast wave-heights. The number of inputs and outputs depends on the physics of the problem and has to be studied in each case. The number of hidden neurons also depends on the problem and its level of complexity. All of them were usually optimized by trial-and-error analysis.

From all the algorithms presented, the BP and the LM were the most used to predict wave height. The efficiency of each algorithm depended on the characteristics of the problem. To choose the best algorithm, each case has to be studied (Fig. 2).

1. Download : Download high-res image (57KB)
2. Download : Download full-size image

Fig. 2. Frequency of algorithms used in wave height ANNs studies.

With respect to the error measures, 22 out of 24 studies used the correlation index (R) as one of the main variables to optimize the network, 65% of the studies also used the RMSE, and the rest of statistics to quantify the performance of the NN depended on the study and on the authors choice.

2.2. Prediction of water level, tides, extremes and wave power

ANNs have also been used in ocean engineering to predict water level, tides and wave power. Tsai and Lee were one of the first ones to use ANNs to predict tides in 1999 (Tsai and Lee, 1999). Lee et al. also used it two years later, they tried different networks with 0, 1 and 2 hidden layers and concluded that the best performance was given by one hidden layer with BP algorithm (Lee and Jeng, 2002; Lee et al., 2002). In 2004 Lee conducted a similar study, but he did it to predict long-term tides, they proved that ANNs were as accurate as traditional harmonic analysis and needed less data (Lee, 2004). In 2006 Chang and Lin compared the ANN results with the NAO.99b model and concluded that both methods were accurate and that the NNs could be applied to different locations with similar tide type (Chang and Lin, 2006).

In 2008 Liang et al. performed one-step and multistep methods, concluding that the last one was better to predict water levels (Liang et al., 2008). This same year, Han and Shi predicted water levels and pointed out the usefulness of this technique not only for short-term forecasts, but also for long-term (Han and Shi, 2008). Next year, de Oliveira and Ebecken used a RBF NN and a GRNN to predict sea level and surge. The results indicated that NNs could also be useful as a complement for the standard harmonic model (de Oliveira and Ebecken, 2009).

In 2010 Wenzel and Schröter used ANNs with two main purposes: filling data gaps in the tide gauge time series and estimating the evolution of regional mean sea levels from these tide gauge data (Wenzel and Schroeter, 2010). Next year, Pashova and Popova compared the performance of five different neural networks on predicting water level and concluded that the best performance was given by the FFNN with LM, LM algorithm was the fastest, and BFGS gave the best results in most cases (Pashova and Popova, 2011). In 2012 Filippo et al. also concluded that ANNs were a very interesting tool to predict water levels and also to complement traditional harmonic analysis approach (Filippo et al., 2012). In 2014 Karamouz et al. used two innovative models called ELMAN (special type of the common FFNN with context units) and IDNN (type of FFTD) for sea-level simulation and saw that sea surface temperature was an important variable to consider (Karamouz et al., 2014). This same year Kusnandar et al. estimated the mean sea level change by hybridizing exponential smoothing and a NN, which gave promising results (Kusnandar et al., 2014). Castro et al. also studied the wave energy potential in nearshore coastal areas using ANNs and saw that the most important variable to do these predictions was the wave-height (Castro et al., 2014).

In 2015 El-Diasty and Al-Harbi compared the performance of an ANN; traditional harmonic analysis and a wavelet network, which is an algorithm that connects the NN with the wavelet decomposition. Results showed that the wavelet network was the most accurate (El-Diasty and Al-Harbi, 2015). In 2018 El-Diasty et al. did another research using wavelet technique, but this time hybridizing it with the harmonic analysis technique, which gave even better results (El-Diasty et al., 2018). In 2019 Zhao et al. applied different ANNs to forecast sea level anomalies, they predicted sea level variation with the least-square fitting methodology and the stochastic residual terms with different NN models. They found that this hybrid approach gave results with relatively high precision (Zhao et al., 2019).

Wang et al. also used wavelet networks to predict tides and sea water level (Wang et al., 2020a, 2020b, 2020c). In 2020 they combined wavelet decomposition and ANFIS models for predicting multi-hour sea levels and highlighted the advantages of these hybrid models (Wang et al., 2020c). After, they improved wavelet networks adding a momentum term (Wang et al., 2020a). This same year, they also improved wavelet networks using a genetic algorithm to optimize the weights and parameters of the NN (Wang et al., 2020b). All these models performed significantly better than the traditional harmonic analysis. In 2020 Avila et al. used Fuzzy systems and ANNs to predict wave energy and both models gave very accurate results without needing a base data of many years (Avila et al., 2020). In this same year Bruneau et al. studied sea level extremes with a deep NN, they used a total of 600 tide gauges around the world to predict the non-tidal residual as well as extremes events and concluded that ANNs outperformed predictions based on multivariate linear regressions (Bruneau et al., 2020).

In 2021 Primo de Siqueira and Paiva used a combined approach of process-based and data-based model to improve the forecasting of sea levels, they used the process-based model to forecast the linear part and the ANNs to forecast the non-linear part and the results obtained were significantly better (Primo de Siqueira and Paiva, 2021). Yang et al. used deep neural networks (NNs with more than two hidden layers) to predict tides and analysed the effects of gaps in data records, they concluded that, among all the methods, deep neural networks performed the best (Yang et al., 2021). Bai and Xu compared RBF and LSTM NNs with an innovative approach: bidirectional LSTM NNs, that is a LSTM NN that combines the information for both, front and back directions of time series, and proved that was the best option (Bai and Xu, 2021). Bento et al. concluded that in the case of extreme unforeseen events (long-term forecasts), the DNN presented a slightly underestimated time-series prediction and more data was needed. The errors of DNNs also depended on the seasonality, winter forecasts were less accurate than summer forecasts due to its greater variability (Bento et al., 2021).

From all the type of networks, the FFNN was clearly the most used. Although in the last years wavelet neural networks are getting more popular (Fig. 3).

1. Download : Download high-res image (54KB)
2. Download : Download full-size image

Fig. 3. Frequency of networks used in sea level, energy and extremes ANNs models.

Referring to the network architecture, the conclusions are the same as in Section 2.1.

From all the algorithms presented, the BP was the most used to predict sea level. The efficiency of each algorithm depended on the characteristics of the problem. However, LM algorithm seemed to be the most effective basing on the studies above presented, the majority of them selected the LM algorithm after proving that it was the most effective (Fig. 4).

1. Download : Download high-res image (54KB)
2. Download : Download full-size image

Fig. 4. Frequency of algorithms used in sea level, energy and extremes ANNs models.

With respect to the error measures, 18 out of 22 studies (which means more than the 80% of the studies) used the correlation coefficient and the RMSE as the main statistical variables to evaluate the performance of the model.

2.3. Application to breakwaters, ports and pipes

One of the first applications of ANNs to maritime and ocean engineering was to study breakwater stability and one of the first authors to do it was Mase, who in 1995 used a 3-layer FFNN to estimate the artificial armour layer stability (Mase, 1995; Mase et al., 1995).

In 2005 Yagci et al. used ANNs and fuzzy logic and saw that ANNs can be employed successfully to estimate the breakwater damage ratio (Yagci et al., 2005). This same year Kim and Park compared the performance of 5 different ANNs for stability prediction (Kim and Park, 2005). They concluded that any NN became more effective when using the design parameters as independent network input rather than using artificial parameters. In 2014 Kim et al. highlighted the convenience of including tidal level in this type of studies to not overestimate breakwater damage (Kim et al., 2014).

In 2008 Iglesias et al. developed a virtual laboratory to estimate breakwater stability (Iglesias et al., 2008). They called the model “a virtual laboratory” because it was capable of estimating the damage caused to a rubble-mound breakwater by waves like a conventional laboratory would do. Two years later, Balas et al. studied breakwater stability with ANNs, fuzzy systems, ANFIS models and ANNs with Principal Component Analysis (PCA) and highlighted that hybrid models performed better than conventional ANNs and captured better nonlinearities (Balas et al., 2010). In 2012 Koç and Balas conducted a similar study in which they used genetic algorithms (GA) and fuzzy NNs, and pointed out that hybrid models needed less data samples (Koç and Balas, 2012). In 2011 Akoz et al. stated that ANNs were an important tool that can be applied for the design of coastal structures under the action of most critical loading (Akoz et al., 2011). In 2013 Suh et al. combined this soft-modelling technique with SWAN model to reduce the computation time and analysed the effects of seal level rise and wave height increase on caisson sliding (Suh et al., 2013).

Referring to application of ANNs to ports studies, one of the most important authors are López and Iglesias, who in 2013 and 2015 conducted two different studies to estimate energy and operability conditions in a harbour (López and Iglesias, 2013, 2015). In 2013 they estimated the amount of energy inside the harbour and they used two different NN models, one that directly estimated the energy and another model that did it in two steps. They concluded that ANNs constitute a valid, efficient approach for estimating wave energy inside or outside a harbour and that the direct approach outperformed the model with two steps (López and Iglesias, 2013). To assess harbour operability, they followed a similar methodology and stated that one-hidden layer NN performed better than 2-hidden layer NN for this case, they also carried a sensitivity analysis and saw that wave height and period were the two variables which most influenced wave agitation (López and Iglesias, 2015).

Previous to these studies Londhe and Deo conducted a research on wave tranquillity inside a harbour (Londhe and Deo, 2003). They used different training algorithms: GDM, RP, CGF, CGP, CGB, SCG, BFGS, OSS, and LM. All the algorithms gave satisfactory results. The CGF algorithm was more memory efficient and the SCG algorithm required the minimum time to train. Kankal and Yüksek. also assessed harbour tranquillity using NNs; they studied the effect of different wave parameters, estimated wave height in the harbour basin and optimized the port layout (Kankal and Yüksek, 2012). In 2020 Zheng at al. used ANNs to estimate waves inside a port and compared it with Boussinesq model performance, they trained the network using numerical models and validated it with observations, proving that ANNs are an efficient tool to forecast wave parameters (Zheng et al., 2020).

In addition, in the last months (March and April 2022) Moslemi et al. have published two articles about the thermal response in pipes containing fluid and gas which has a great application in ocean engineering (Moslemi et al., 2022a, 2022b). In the first study (Moslemi et al., 2022a) they used ANNs to determine the temperature and phase change rate in the extrusion process, they compared the experimental and numerical results and concluded that numerical results can be used in ANNs training and that ANNs can be used to optimize the parameters of additively manufactured polymers used in ocean pipelines. In the second study (Moslemi et al., 2022b) they applied a FFNN to determine the correlation between the welding pool characteristics as input and the Goldak parameters as output and concluded that ANNs outperform regression models, but they are more complex.

From all the type of networks, the FFNN was clearly the most used (Fig. 5).

1. Download : Download high-res image (58KB)
2. Download : Download full-size image

Fig. 5. Frequency of networks used in breakwaters, ports and beaches ANNs models.

Referring to the network architecture, the conclusions are the same as in Section 2.1 Prediction of wave heights, 2.2 Prediction of water level, tides, extremes and wave power.