The effects of turbulence generated by nonbreaking waves have been investigated by testing and evaluating a new nonbreaking wave parameterization in a coupled hurricane‐ocean‐wave model. The MPI version of the Princeton Ocean Model (POM) with hurricane forcing is coupled with the WAVEWATCH‐III (WW3) surface wave model. Hurricane Ivan is chosen as the test case due to its extreme intensity and availability of field data during its passage. The model results are validated against field observations of wave heights and sea surface temperatures (SSTs) from the National Data Buoy Centre (NDBC) during Hurricane Ivan and against limited in situ current and bottom temperature data. A series of numerical experiments is set up to examine the influence of the nonbreaking wave parameterization on the mixing of upper ocean. The SST response from the modeling experiments indicates that the nonbreaking wave‐induced mixing leads to significant cooling of the SST and deepening of the mixed layer. It was found that the nondimensional constant b₁ in the nonbreaking wave parameterization has different impacts on the weak and the strong sides of the storm track. A constant value of b₁ leads to improved predictions on the strong side of the storm while a steepness‐dependent b₁ provides a better agreement with in situ observations on the weak side. A separate simulation of the intense tropical cyclone Olwyn in north‐west Australia revealed the same trend for b₁ on the strong side of the tropical cyclone.