Concern about the inflationary impact from tariffs has eroded confidence among both businesses and consumers, according to recent surveys.