Breach of trust
Trustee duties are not aspirational. Acting outside the governing document, applying funds to non-charitable purposes, or failing to act with reasonable care can all amount to a breach of trust.
Where a breach causes loss, trustees can be required to make good that loss personally. The courts and the regulator have powers to relieve trustees who acted honestly and reasonably, but relief is discretionary, not automatic.
The practical protection is process: act within your powers, take advice where you should, and record decisions. See trustee duties, and note that indemnity insurance does not cover everything.