Thematically, and although slavery rightfully has a very bad rap, It fits way better to enslave the minions of the ennemies of Odin than to be helped by humans that supposedly have very little to do in Valheim next to our (fallen?) hero/team of heroes.
I play the game solo. The loneliness can sometimes be… heavy, but it's also part of why the game is so enjoyable. You're supposed to be a sole viking or a small crew of people doing the fight for Odin.
I think that human NPCs, even if they are minions rather than actual NPCs with dialogue lines, would negate a good part of the charm of the game. I'm personnally not interested in having the equivalent of Minecraft villagers.
Having a limited number of minions, enslaved from the forest, would make more sense. Having a few greydwarf minions around is, IMO, way less of a problem thematically and atmosphere-wise. It would of course be mostly a near-endgame feature, because you only really need it once you're crawling under ressources / ressource farms to manage.
It would better explain their lack of involvement in the actual task ahead. Being very limited beings, they could do only menial tasks that are mostly cores to most players (think of resin feeding the torches for example…), which would make more sense than human NPCs not taking up arms to help you.
If you absolutely hate the idea of a viking slaver, well… just don't do it I guess, same as me with villagers if they ever become a feature.
Continuing on this idea, if people are really getting angry about the slavery thing : greydwarves formerly being terrible people punished for their deeds, we could try through some work and special rituals to redeem some of them. You could go the slavers' route, or actually care and try to help them redeem themselves in the eyes of Odin through their help of the player.
TL:DR : Greydwards are little shits that I would like to enslave, which IMO would better fit the themes of the game (dominating a land and the creatures which seek revenge on the Old One) and would have less impact on the overall atmosphere of loneliness/solitude of the game.