The number of drivers can impact the time it takes to run, but shouldn't have the largest impact. Information about the drivers is compiled into the HII driver library database, which is transferred to the client. The client then goes through the DB to find matches, then downloads those matches. Once they are downloaded, they are added to the Windows driver store with DISM. Looking through the DB is normally very quick. If you look at the HIIClient.log you might get some insight into what is taking a long time. Look for messages about when it downloads the HII database. How long before it is done or moves on? Is the matching part taking a while? My best guess would be that it's the final step that is taking so long, which is downloading the actual driver files, and you should be able to see that at the end of the log. Maybe the number of drivers is somehow making the source deliver files slower?
As to cleaning up duplicates, I don't know a sure-fire way, but one option might be to run hiiclient.exe /preview on your various hardware models. That will tell you which drivers would be matched on that hardware. Compile a few of those together, then delete the rest?
Oh, and the HIIClient.log will normally be on machines in C:\Windows\Temp after they have been imaged.