Are there downsides to not caring for my Pokemon after a battle?

No, there are no downsides, per se. There are just no upsides.

If you do choose to care for them, you will get more buffs as Pokemon have more Affection towards you. A full breakdown of these can be seen in this Q&A.

If you care for your Pokemon after battle, they will have a chance to do things like survive a hit that would have otherwise knocked them out, be more likely to get a critical hit, or dodge an attack. Thus, it's recommended to spend the ~30 seconds brushing your Pokemon after a battle if you receive the prompt.

The after-battle treatment screen is actually Pokemon Refresh. Everything you can do in Refresh, you can do in that screen, with the addition that if your pokemon have taken a certain level of damage or have been hit with a status condition, you can clean them up, dry them off, apply medicine, ect. to give them additional affection.

In Refresh, caring for your pokemon increases its affection for you (note that this a different stat than friendship). A pokemon starts with 0 hearts of affection for you, and this can increase up to a maximum of 5, with each heart of affection after the first giving you additional buffs (such as additional experience, better critical hit ratio, a chance of a focus sash-like effect, ect). After maxing out a pokemon's affection, subsequent attempts to pet or feed the pokemon do not have any additional effect (besides looking adorable); this applies to the after-battle care tools as well. The only exception is the medicine (which looks like a pair of tweezers holding a cotton ball). If your pokemon has a status ailment that persists outside of battle, such as sleep, paralysis, or poison, using the medicine on the pokemon will remove that ailment, saving you money on full heals/antidotes/ect.

Also important to note; if you plan on playing in matches against human opponents or participating in the Battle Tower, note that you can't use the buffs from Refresh.