Intercity buses connect all of the major (and many minor) towns and cities in Taiwan. The buses are run by several different companies so the prices and comfort levels vary widely.
If you are leaving from Taipei, prices are similar to that of the standard trains, so unless you are traveling to somewhere without a train connection, you’d probably be better off on the train.
If you are planning on taking buses to the more remote areas, make sure to find out when the return journeys are. Many of these buses only leave a few times a day and you don’t want to get stuck without transport somewhere!