Is it illegal for my neighbor to open my back door that is unlocked and let my dog outside?
Yes. It is trespass. Also perhaps malicious mischief, if your dog gets out and runs away. Also perhaps theft, if your dog runs away. If your neighbor intended to commit a crime in your house, he has also committed burglary.