Will a track system make my ATV/UTV more likely to tip?

No. Although you gain about 4 inches of vehicle clearance with a track system installed, the additional weight near ground level and the longer, wider wheel base actually increase stability.