My humble opinion based on that short walk from a popular Hollywood watering hole where celebrities go for a drink without anyone being allowed to bother them: Yes, its possible they had a rendezvous and he failed to get his shirt tucked in properly afterwards. Yes, its possible they could have been on a date and are heading out for dinner at The Ivy after drinks at Chateau Marmont and he's simply not paid much attention to the condition of his clothes after a visit to the men's room. Yes, they could have been attending a wrap party for Legends at Chateau Marmont. Yes, they could have become acquainted during GoT filming in Ireland and met up as just friends out for an evening in LA before he heads back to UK. Yes, he placed his hand on her back to guide her into the limo first, just like he always does when accompanying ladies just about anywhere. No, he did not hold her hand as they walked down the drive to the waiting limo, like he normally does when he's out with a lady he is more than just friends with. No, neither one of them are attempting to avoid publicity or trying to hide that they are together. Yes, Esme may have gotten in touch with him while he was in LA to see what he could do for her as far as getting work in the industry in Hollywood. Yes, he may have rung her up and arranged to meet because she's his type and he's got an itch to scratch (the man is not a monk, afterall). Yes, there are just way to many variables to make any logical assumptions with just this few seconds of footage.
Now, given all that, if there are more tweets in the near future of them being spotted together or more instances of them being photographed or videotaped together I may reassess my opinion.
And by the way, in 2012 she "moved to Los Angeles to pursue her growing acting career".