There have been more than a few bad hat decisions lately (Yes, we’re talking about you, Pharrell) and, sadly, not too many good ones. There are beanies with fake beards and flat-billed caps. What’s worse, people actually wear them. People wear fedoras with t-shirts, baseball caps sideways—it’s really not hard to get this right but it’s apparently pretty easy to get it wrong too.

Before you order a hat, or even try one on, you should know how this works. Here’s a simple guide, a few notes on how to wear a hat the right way.

With a Suit
Whether you’re on your way to a business lunch or your great aunt’s funeral, a hat’s a great way to add a little something special to your dressed-up look. Make sure to leave the baseball cap at home unless you’re taking a lunch break to watch your son’s little league game.

Instead, don a fedora or panama hat in a color complementary to that of your suit. If you’re new at it, wear it straight but, once you’ve settled in, a slight angle and a little bit of a downward tilt works but you have to have the personality to back it up.

With a Casual Shirt
You can go either way here. Wear a baseball cap (with a slight curve to the bill and only forward or, if you really must, backwards) or a beanie if you’re activity warrants dressing up from a t-shirt. Wear a fedora or panama hat if you’re dressing down from a suit and tie.

The type of shoes you put on can tell you which way to go too. Don’t wear a fedora with flip flops and don’t wear a beanie with dress shoes. A straw hat’s the exception—this can be worn with anything from flip flops to patent leather.

With a T-Shirt
You have three choices here—a straw hat works if you’re going to be outside, especially at the beach, while a baseball cap or beanie works pretty much the rest of the time.

If you’re still not sure which one to wear, reach for a straw fedora—it’s the most diversely wearable hat around with enough class to match a great suit but it’s relaxed enough to look right at home on the beach.

With a Coat
The coat itself doesn’t change anything—it’s what’s underneath that matters. If you’re wearing a suit, follow those rules. If you’re wearing a casual button-up shirt, follow those. If you have a t-shirt underneath, stick with a baseball cap or beanie but do it the right way.

Just as a sidebar though, wear the right kind of coat. You shouldn’t wear an overcoat or trench coat with sweat pants and a t-shirt any more than your puffy jacket goes with a suit. It’s a whole different set of rules but make sure you have those down before you ever drop a hat on your head.