Hi MrGadget,
I have the same issue.
Just to be sure about the driver downloading. Did you enable the XTrace in your WinPE ? See this document How to enable XTrace Diagnostic logging in LANDesk Core and Clients
Enable the log for lddwnld and tell us the error code when the agent try to connect to your preferred server.
Did you apply the LD-CP-BASE-2014-0417 on you core server ? If you use a WinPE 64bits it fixes a maptoprefferredserver issue.
I'm still trying to debug this issue on my side.