They both sound natural, but I'd say there's a subtle difference... Maybe other native speakers could weigh in to make sure this difference isn't a regional thing. To me, the phrase "to your left" means "about ninety degrees to the left of where you're facing," and the phrase "on your left" means more generally "to the left of where you're facing." In your example, if we were walking down the street, I'd probably know exactly which building you were talking about if you said "the building to your left," but I might ask, "Which one?" if you said "the building on your left." You might be referring to a couple different buildings. I'd also say there are subtle differences when you further modify "on/to your left." For example, it sounds much better to me to say, "the next few buildings on your left" than "the next few buildings to your left." Similarly, if you said "all the buildings to your left," I'd interpret that to mean the buildings that I would see if I turned ninety degrees to my left, possibly including buildings behind me. If you said "all the buildings on your left", I'd interpret that to mean the buildings in front of me on the left hand side of the street.