Removing an NSX-T Manager from the Manager Cluster is actually quite simple.
Go to System > Appliances, select Action and then Delete on the appliance you want to delete.
But what if it is the first appliance that was manually installed? In this case, deletion via Action is not possible. How it can be done is shown in this article.