In 1831 there was a crisis between the settlers in Port Natal (Durban) and Dingane, the Zulu king. Some settlers boarded a ship that was in the harbour and the others, including Henry Francis Fynn and his family, fled down the coast. The Zulu warriors caught up with them where Port Edward is today and massacred the fleeing settlers, which included local tribespeople of Langeni, on a hill called Isandlundlu (in English, shaped like a hut). The place has been known ever since as Tragedy Hill and its slopes are still littered with the bones of the victims.

In 1552, the Portuguese carrick "Sao Joao" ran aground at Port Edward and this is the first time in recorded history that peoples from Europe met peoples from South Africa.

In 1878, the ship "The Ivy" ran aground on Leisure Bay area beach.

In 1925, the area was partly owned by TK Pringle, and he named the inland portion Banner Rest as this was where he wished to "strike his banner". The village was laid out and was named Port Edward in honour of the Prince of Wales, who later became King Edward VIII.