I've seen this wonderful old watchtower from River Silver quite a few times and made up my mind to explore it when I had more time as it looked quite Famous Five-ish.
After seeing it up close, I did some judicious Googling and found out that it's called the Chung Hau old mansion. It was built out of cut granite blocks sourced from the hills around the area in the 1920s by Yuen Kwan-yu as a guard tower as well as protection against pirates.
The old tower is still owned by the Yuen clan, which has modern houses in the family compound. There even is a cannon (stopped up with cement now) guarding one of the buildings.
You can read more about the history of the watchtower here.