Setting up the core server as a preffered server did the trick. I read in this discussion that its possible to do. I just added the core server using a dummy IP range (1.1.1.1 to 1.1.1.2) as mentioned in the discussion.
Just to note for others running into the same: I did enter in some read-ony credentials when setting it up, which I know for sure have access to the core shares. But when I clicked the Test Credentials, it would fail for both UNC and HTTP. However, the deploy image action still seems to work (weird). I'm guessing as long as its listed as a preffered server, then it works??
Also: I did not setup any of the other settings for the core/preferred server (Selected Replicator, Sources, or Write Credentials)