I don't worry about it too much outside of my profession. Writing and spelling English is not easy, and some people really do struggle, like I struggle with doing math in my head.
When you take dialect into consideration, it's easy to see why some people struggle more with there/their/they're and your/you're. Where I come from, most people pronounce those words differently. In very large parts of the United States, they're pronounced exactly the same. So there's no "hook", so to speak, to hang the spelling on, and not everyone is good at rote memorization.
I remember in school struggling with "warranty" and "guarantee" because here they sound sufficiently alike and mean nearly the same thing.