sure you can demolish all housing then, loyalty will stay the same and income from commercial too. Only your population will drop, which results in a decrease of tax income but also a decrease in food needed.
Do note that if you cluster that colony you need population to build more infrastructure on the new cluster. Also don`t kill all housing but leave one housing on the planet, that way you can always decide to repopulate it.