Punctuality

I have a friend who is always late and usually by quite a bit (20 minutes or so).
When I'm late for some things it is super stressful for me, but other things I actually like being a little late too. Also, some things I find it easy to be on time for while others I always feel like I'm almost always accidentally late. So what is the difference?
Things like church and classes I'm taking, I find it relatively easy to be on time for (and I hate being late to). These things are consistently at the same time and I usually can gauge the exact time it will take for me to get there because all it entails to get there is to walk there from my current location (which is usually the same place).
Social engagements especially with larger groups of people I sometimes prefer to be late (5 to 10 minutes) because that way you aren't awkwardly early and it is easier to slide in unnoticed. But half the time I am more late than that. I think my lateness usually derives from being overly optimistic about the time it will take me to get somewhere. This is especially problematic when I'm in the middle of something else, because it is easy to say "oh I can just finish this one thing and still get there. It won't take me that long."
Unfortunately, it is the unusual appointments with only 1 or 2 people that I get most stressed about when I am late, but are also some of the most difficult for me to get to on time because I am likely to already be involved in other things before them and I'm also probably not used to going wherever it is so I will be overly optimistic about the time it will take me to get there... A rather unfortunate combination.