Evaluating the effect of volatile federated timeseries on modern DNNs: attention over long/short memory

In regards to the field of trend forecasting in time series, many popular Deep Learning (DL) methods such as Long Short Term Memory (LSTM) models have been the gold standard for a long time. However, depending on the domain and application, it has been shown that a new approach can be implemented and possibly be more beneficial, the Transformer deep neural networks. Moreover, one can incorporate Federated Learning (FL) in order to further enhance the prospective utility of the models, enabling multiple data providers to jointly train on a common model, while maintaining the privacy of their data. In this paper, we use an experimental Federated Learning System that employs both Transformer and LSTM models on a variety of datasets. The sytem receives data from multiple clients and uses federation to create an optimized global model. The potential of Federated Learning in real-time forecasting is explored by comparing the federated approach with conventional local training. Furthermore, a comparison is made between the performance of the Transformer and its equivalent LSTM in order to determine which one is more effective in each given domain, which shows that the Transformer model can produce better results, especially when optimised by the FL process.