You are leaving the province. It's utterly absurd that a day or more
away fromt eh place you just left, you get notice that someone on your
hostile list showed up and you fly back, waste him and return to your
place in departing, having lost only the day in combat.
If the hostile will catch people in a province when you are N days out of
it, then the only thing that begins to come close to even handed is that
you _at least_ lose all movement that you've accrued in making the
attack, and have to start over.
However, I think that it should be a case of you're leaving and you
missed him. Otherwise, we make provinces even more of a barrel in which
to shoot fish....