OT3 in Canon
Jack. Kate. Sawyer. Clearly the love triangle of Lost. Hero and bad boy fight over the girl -- only she ends up being badder than both of them -- a scenario custom-made from JJ Abrams (or Joss Whedeon or James Cameron).
I've heard that JJ Abrams didn't plan on throwing Sawyer and Kate together until he saw the chemistry had together in the "I know your kind" scene. Well, I buy unexpected chemistry as the reason there's a Shannon/Sayid romance but really -- every JJ Abrams show is built around a triangle: Noel/Felicity/Ben in Felicity, Vaughn/Sydney/Will and later Sydney/Vaughn/Lauren in Alias. The man just loves triangles!
I think we all know how the triangle works in canon. Kate pursues Jack, he pulls away. Sawyer pursues Kate, she head butts him. An oversimplification, yes, as are their archetypes: Jack is the noble leader who cares too much, Sawyer the brooding man with a secret and Kate the mystery woman with a past.
A prime example for me of the shifting dynamic is how Kate tries to play them both in "Whatever the Case May Be" until it looks like Jack and Sawyer might start trusting each other more than they trust her. Sadly, that scenario didn't play out due to Jack's shocking inability to hold a grudge against anyone but Locke.
OT3 in Fanon
But let me get to the gist of my post: the OT3 in fanon. (Spoilers for several fics below - I'm going to assume you've read all of these before.) I'm not going to touch on every JKS fic, just the ones that are the most typical or marked a shift in the way I saw them and their dynamic. I'm addressing these more or less in chronological order.
The first OT3 fic I read was Only Natural by cmere (and join me in lamenting that she hardly ever writes Lost fic. Sigh.) It's told from Jack's POV and Kate and Jack are finally going to have sex when Sawyer shows up and Kate wants to include him. And Jack, surprisingly agrees. And is even more surprised to find that Sawyer is just as interested in him as he is in Kate. When it's all over, Jack can't let go. For me, this set up a lot of other OT3 fics where Jack is the most vulnerable of the three. The other two are adept at letting go (something we all know Jack isn't good at) and letting sex just be sex.
Another OT3 fic with similar characterization is Rewind by meltyoudown and also from Jack's POV is, where he hates himself for being part of the threesome and gives Kate an ultimatum: Him or Sawyer. Here he's not willing to admit there's an attraction between himself and Sawyer, to his own loss. He does come off as a bit of a prude -- or they're completely degenerate, depending on how you want to look at it.
Then foxxcub went and changed the whole game by writing Ballsy!Jack in Quid Pro Quo (Sawyer POV). He was forceful, dynamic and fucking bossy and damn if that didn't turn my whole Lostverse upside down! Up until then whenever I wrote Jack/Sawyer, Jack had a big ol' bleeding heart but her fic made me see him in a whole new light. But now Jack could be calling the shots and ... wow! (It also brings us Voyeur!Kate, which isn't a bad thing either. ;-D)
Because of that fic -- and how we saw a new, maniacally driven side of Jack in "Do No Harm," I wrote Possession and Blackout (both Kate POV) in which Jack is very, very dark and holding Sawyer and Kate in kind of sexual thrall. So by default, Sawyer and Kate become much more vulnerable. I was also interested in exploring Jack and Sawyer as the main couple, with Kate as the outsider.
desire_of_mind wrote The Devil Only Smiles, in which Sawyer is concerned that Kate is using Jack and that Jack is going to be hurt. It's interesting for a number of reasons. One being that Sawyer seems to be the only one to see how manipulative Kate is and also that Sawyer's sympathies lie with Jack, making this pre-slash. Again, Jack here is the innocent who's going to get hurt by tangling with either of them, it seems.
eponine119 wrote Jack Hates (Jack POV), a future fic in which Jack and Kate are married and he's an alcoholic and things have deteriorated completely because Jack can't get over his jealousy of her previous relationship with Sawyer. So he leaves to confront Sawyer but realizes that this whole time they were both just using Kate when they really wanted each other. And this fic kills me because here Jack is facing the truth for the first time and he can't handle it. I guess I always see Sawyer or Kate as the ones with the walls that can't be torn down and the inability to love or trust but when it's Jack that's so much more devastating, to me. It's absolutely still true to who he is and so credible, but still completely heartbreaking for Jack to be the one to pull away.
For a fic where Sawyer's left out in the cold, by his own choice, see Shake It Off by foxxcub, in which Sawyer dares Kate to go after Jack, much as he dared her to kiss him in foxxcub's earlier Sawyer/Kate fic, Gin and Gemini. And since the J/S OTP had also claimed foxxcub by then, Sawyer finds himself more turned on by the idea of Jack, then of Kate.
(And speaking of J/S shipping, here's a few fics that are basically J/S with a side of Voyeur!Kate, I Never Until Now by kho and two drabbles by lillyjk: Here and here.)
As a request, I wrote Turn My Way (Sawyer POV) for uberaeryn where Jack and Kate are the main couple (which seems to be where the show's creators are headed, eventually, or so they tell us), and Sawyer is jealous of them. But she wanted both Jack and Kate to have feelings for Sawyer. In that, I really wanted to deal with Sawyer's guilt over taking up his role as homewrecker again - something he'd given up prior to the island. That ended up becoming a more minor aspect of the whole fic because it was getting rather OOC.
uberaeryn wrote from Kate's POV in Walls. And it's fucking brilliant, even more so in that it's her first stab at writing Kate at all. In it, she is tired of being the buffer between Jack and Sawyer, the only thing that keeps them from beating each other to a pulp. So one night she just strips off her clothes and everything happens wordlessly and inevitably. She realizes they both need her and that through her they're also making love to each other. I'm not really doing it justice by paring it down to plot basics, but you must read it. (I'm gushing a bit more on this one because it's still so new and shiny, where as the others are old, beloved favorites.)
So, anyway, I don't have a conclusion, just that I've enjoyed seeing how this OT3 has evolved, both on the show, where Jack and Sawyer have become much closer and more similar, and in fandom, where new fics can put a whole new perspective on the characters of Jack, Kate, and Sawyer and how they interact. Obviously some of those seismic shifts are happening in J/S or S/K or even J/K fics too, but I'll save that for a different post. ;-D