I add all of the drivers except one video driver. It's really not that many since Windows 10 has a lot of the existing older hardware covered with default drivers in the OS.
For example, the only model of hardware that I need to add the intel chipset driver is the HP Z240s we use. Someday a newer version of Windows 10 (1803?) will have that inside as default driver so I would remove it and will then probably have to add the Z250 chipset driver when it replaces the Z240, and so on.
The only driver we need to add post-build is Matrox video and that gets installed by its graphical software package (Powerdesk), which we have to install and keep up-to-date to resolve issues. So it doesn't make sense to maintain that in the image because it's software (+driver).
I also use a graphical version of dism I wrote in powershell which makes mounting/unmounting the image and adding/removing driver packages really easy. But honestly, I don't think I've touched the image in over 6 mo.