There are currently 122 heroes, so the number of ways to pick two unique teams is:
122C5 x 117C5 = 207,288,004 x 167,549,733 = 34,731,049,724,302,932
We get the final answer by multiplying this number by 2 because we consider Radiant and Dire as unique sides:
34,731,049,724,302,932 x 2 = 69,462,099,448,605,864
Rounding up, the number of Dota matches played to this date is 6.5 billion. Assuming all these games have unique hero compositions (which is unlikely the case), we can divide 6.5 billion by the total we have above to get:
6,500,000,000 / 69,462,099,448,605,864 = 0.0000000936 = 0.00000936%
Which means even if all the Dota games ever played have unique hero combinations, we have only played 0.000009% of all possible combinations.
If we continue playing one unique game per second, it would take:
69,462,092,948,605,864 / 60sec / 60min / 24hrs / 365.25days = 2,201,121,107 years
Over 2 billion years to have played one match of each combination of heroes.
I hope the minute that you have spent reading this was a nice distraction from the state of the world we currently live in.
Edit:
I think my rate of 1 game per second seems too low, since we played roughly 6.5 billion games in 10 years, which would put the rate at 650 million games per year, about 21 games per second. Using this new rate, to play one match of each unique combination of heroes, we need another 104,815,290 years of playing 21 games per second.
Edit 2:
Thanks to u/Pkris04 for pointing out, I shouldn't have multiplied 34,731,049,724,302,932 by 2 because all possible combinations for both teams were already covered. Another way of arriving at the same answer would be to take the number of ways to pick 10 heroes from a pool of 122 and multiplying that by the number of ways to pick two unique teams of 5 from that chosen 10:
122C10 x 10C5 = 34,731,049,724,302,932
This would bring the number of years down to 52,407,645 years to play all the combinations if we continue playing at 21 games per second.
Source: https://www.reddit.com/r/DotA2/comments/t2q1lt/lets_do_math_counting_the_number_of_unique_dota/