Close your fireplace flue when you are not using the fireplace. It is a big hole that runs up your chimney to the outside. It is designed for upward flow, and that is what it does even when there is no fire burning.
If your fireplace has a built in fresh air intake, make sure it is closed when the fireplace is not being used. Close the fresh air intake even if your fireplace has glass doors. These glass doors are not designed to insulate from cold outside air. Close the fresh air intake and the fireplace flue when not in use. You may not feel cold air coming down the chimney, but warm air is most likely escaping up it.