As mobile internet and Internet of Things technologies continue to advance, the application scenarios of peer-to-peer Internet of Drones (IoD) are becoming increasingly diverse. However, the development of IoD also faces significant challenges, such as security, privacy protection, and limited computing power, which require technological innovation to overcome. For group secure communication, it is necessary to provide two basic services, user authentication and group key agreement. Due to the limited storage of IoD devices, group key negotiation requires lightweight calculations, and conventional schemes cannot satisfy the requirements of group communication in the IoD. To this end, a new lightweight communication scheme based on ring neighbors is presented in this paper for IoD, which not only realizes the identity verification of user and group key negotiation, but also improves computational efficiency on each group member side. A detailed security analysis substantiates that the designed scheme is capable of withstanding attacks from both internal and external adversaries while satisfying all defined security requirements. More importantly, in our proposal, the computational cost on the user side remains unaffected by the variability of the number of members participating in group communication, as members communicate in a non-interactive manner through broadcasting. As a result, the protocol proposed in this article demonstrates lower computational and communication costs in comparison to other cryptographic schemes. Hence, this proposal presents a more appealing approach to lightweight group key agreement protocol with user authentication for application in the IoD.