The research of emotion recognition based on electroencephalogram (EEG) signals often ignores the related information between the brain electrode channels and the contextual emotional information existing in EEG signals, which may contain important characteristics related to emotional states. Aiming at the above defects, a spatiotemporal emotion recognition method based on a 3-dimensional (3D) time-frequency domain feature matrix was proposed. Specifically, the extracted time-frequency domain EEG features are first expressed as a 3D matrix format according to the actual position of the cerebral cortex. Then, the input 3D matrix is processed successively by multivariate convolutional neural network (MVCNN) and long short-term memory (LSTM) to classify the emotional state. Spatiotemporal emotion recognition method is evaluated on the DEAP data set, and achieved accuracy of 87.58% and 88.50% on arousal and valence dimensions respectively in binary classification tasks, as well as obtained accuracy of 84.58% in four class classification tasks. The experimental results show that 3D matrix representation can represent emotional information more reasonably than two-dimensional (2D). In addition, MVCNN and LSTM can utilize the spatial information of the electrode channels and the temporal context information of the EEG signal respectively.