I believe the physical factor is: The difference in Brain.
Even though in book, the Frankenstein is constantly referred to a creature without a heart to feel emotion, the true part of our body that control our judgement and our emotion is our brain.
Each part of our brain will process different types of triggers that will produce the output in the form of emotion and decisions
