Who should cover the loss.

I am a subcontractor that delivered flooring materials to a jobsite at the builders request. The foreman on the job was aware that this was a more than one day job and informed me that it was safe to leave the materials on the site. So after a days work I went home. When I returned the next day about $700 dollars worth of materials were gone. Now the builder is saying that I am responsible for covering the loss, is this correct?