"We're saying the word 'last' a lot more than we ever did," he told PEOPLE on Wednesday during a Television Critics Association set visit. "We knew that it was going to come to an end, but we never really talked about the end, so this season has been filled with a lot of us saying, 'Oh.' "
As Harris and his castmates Cobie Smulders, Josh Radnor, Jason Segel and Alyson Hannigan fielded questions from reporters, both of the actresses said, "You guys really want us to cry."
Sure, it's a cliché but this cast really seems like family, and these actors readily admit that they clicked early on.
"Every Monday for a large part of the first season, we met at a place nearby for breakfast and it was really sweet," Radnor, 39, says.
"At the beginning of this year, I thought that we should meet before the first table read for breakfast at this place, and I sent out an email, and within 20 minutes everyone had responded, so we met and had some food. How is that for a story?"
I'm Going to Miss ..."I'll miss the shorthand," Harris, 40, admits.
"Through the duration of time, we're not only finishing each other's jokes, we're finishing each other's sentences. Like they all know how I react to things. So when I'm upset about something, I don't have to give all the details. I can just sort of start a story and they'll calm me down quickly because they know what's effective for me and vice versa. So we get these half conversations that involve a lot of laughter and calming each other down."
A lot has happened in their lives, both onscreen and off.
"We watched each other go through, like, some really serious life stuff. People have gotten married and have children. That's a really special thing," Segel, 33, says.
"As the characters evolved, all of our lives did as well, and some of it paralleled what we've gone through as people. This is the longest I've known any group of people besides my family … You really do become like family. It's really special."
But he says the fact that the show is ending won't really hit them for some time. "I don't think it will be odd when we wrap," he says. "I think it will be odd when it comes time for when season 10 would start, and you realize you're not going back."
Staying in Touch and Moving to N.Y.C.While they haven't quite made a pact to meet on the third Saturday of every month, Harris is sure that they'll keep in touch as they move on with their lives. When the show wraps, Harris and Smulders, 31, are both moving to New York City, where he will be appearing in Hedwig and the Angry Inch on Broadway and she will be reunited with her husband, Saturday Night Live's Taran Killam.
"I assume that those with kids will probably keep our relationships," Harris says, prompting Radnor to add, "Jason and I are going to hang out a lot."
And when they walk off the set for the last time, each plans to take a souvenir of the past nine years with them – if executive producer Carter Bays doesn't get his hands on it first.
"The last time we talked about this, I realized that Carter is actually taking everything," Hannigan, 39, says. "I was like, 'I want this,' and then he's like, 'No, I've already got that.'"
Smulders says that she'd like to keep her entire wardrobe and a light in the bar that turns red when the bar is closed, but Harris has a more iconic piece in mind.
"I'd love to take the bar table that we have," he says. "We all carved our initials and stuff in it. But egotistically, I think it's going to end up at the Smithsonian. So I might take a stormtrooper from Barney's apartment, maybe if David [Burtka, his partner] will let me put him in the house."