To me, the second half of Season 2 is where the Core 4 relationships really begin to shift.
Xander begins the season supertight with Willow, and distant from Buffy; however, by the end of the year, he is really close with Buffy & Cordelia (to the point where he's essentially supplanted Giles as Buffy's second in command).
Whereas, during that same time period, Willow is mainly preoccupied with Oz & mourning Jenny. She & Buffy are still fairly close, but nowhere near as close as they once were, pre-Innocence imo.
By the time Season 3 rolls around, Cordelia, Xander, Willow & Oz have all seemed to reach a detente in Buffy's absence. After the drama surrounding Buffy's return subsides, Buffy and Willow resume their close relationship (save that brief period Buffy was obsessed with Faith).
Xander & Buffy, however, never really recover. He rightfully questions her judgment while she constantly questions his motives. In the aftermath of the-Fluke, Xander grows even further from the Core Group, as Willow & Buffy, especially, grow much closer. Xander's brief dalliances with Faith & Anya are pretty much disregarded, and he essentially becomes a glorified extra.
And that particular set of group dynamics (B/W,G,X) remains for the rest of the show IMO.