Villagers are a balancing act, for sure. Left alone, they are incredibly overpowered, because they never despawn, can basically breed infinitely, and basically just ruin everything. However, the removal of the shop changes that, as well as the breeding nerf.
The items they trade in are keeping the economy moving. The Ebook market is one of the only industries where people can make a living, currently.
Removing villagers will make the economy dependent solely on what people get for themselves. All money will come in through minigames, and all items through people's own work, rather than purchasing and selling goods.