Mui Wo's forgotten buildings

We were walking past Chung Hau Village when we noticed a few huge buildings tucked away behind the old houses behind the beach. This one looks like it used to be a grand mansion. 

But it is really crumbling down one side.

 This yellow building is even more intriguing. It looks like it used to be shophouses.

If it were shophouses though, there's no easy access. Here's the way we entered.

Any old timers have any idea what it used to be? Here's a view of the yellow building from the road.