The urban metro system accommodates significant travel demand and alleviates traffic congestion. Improving metro operational efficiency can increase the metro operator revenue and promote the development of robust urban transportation. To achieve this goal, passenger flow forecasting is a crucial and well-recognized task in metro operation. However, passenger flow forecasting is a challenging task as there exist many unquantifiable factors in resident travel. To address this problem, we propose an innovative model named Interaction Graph Network (IG-Net) to perform passenger flow forecasting at the station level, capable of capturing the non-Euclidean relationships between stations. Three kinds of inter-station interaction graphs are developed to model these inter-station interactions: connectivity, similarity, and temporal correlation graphs. Moreover, we apply multiple channels of graph convolutional neural networks to capture interaction representations and develop a multi-task learning architecture across multiple stations. The proposed IG-Net achieved better performance than the benchmark models when forecasting passenger flow over multiple stations, based on experiments with the Suzhou metro. Finally, we identify the significant effects of interaction graph combinations and multi-task loss functions via further experimentation.