Deep learning has been used in various research fields. In communications, the deep learning has also been applied to improve the system performance and computational complexity. To jointly optimize transmitters and receivers, an end-to-end autoencoder with neural networks (NNs) was applied in the multi-input multi-output (MIMO) communication systems. This system does not require channel state information (CSI) and avoids the high-complexity maximum likelihood (ML) detection at the receiver. Therefore, this system can be a good alternative to next-generation communications which require high data rates and low latency.
In a half-duplex relay communication system, a source (S) sends signals to a destination (D) through two time slots. In the first time slot, the signals are broadcasted to the destination and a relay (R). In the second time slot, reconstructed signals are transmitted from the relay to the destination. Therefore, it is required to store the reconstructed signals at the relay before transmitting to the destination. An amplify-forward (AF) method, one of the major relay methods, requires the CSI of the source-relay (SR) link and high memory demand because of the amplification of received signals and its continuous nature. On the other hand, a quantize-forward (QF) method does not need any CSIs and only requires small amount of memory.
In this thesis, autoencoder-based QF relay systems including an encoder part (source), a decoder part (destination), and a channel part with the source-destination (SD) and the source-relay-destination (SRD) channels are presented. The QF algorithms at the relay determine the channel part (the SRD channel) and thus greatly affect the system performance. Therefore, various QF algorithms such as the phase quantization (PQ), PQ with NNs (PQNN), amplitude-phase quantization (APQ), and APQ with NNs (APQNN) are proposed. The PQ only quantizes the phase of the received signal at the relay while the APQ quantizes both the amplitude and the phase of the received signal. The PQNN and APQNN apply NNs at the relay and thus the SRD channel will be jointly optimized with the encoder and the decoder. The simulation results show that the APQ and APQNN algorithms achieve much better performance than the PQ and PQNN algorithms. The PQNN and APQNN algorithms may achieve better performance than the PQ and APQ algorithms when there exist a lot of training data or very small amount of training data. Moreover, a sub-message one-hot encoding method is proposed to solve the high-complexity problem for the autoencoder-based QF relay system with large numbers of antennas.