How to make Skyrim more challenging at higher character levels?

Solution 1:

Is there anything that I've done incorrectly for my build, in that I shouldn't have tried to get the best gear midway through the game?

This is very subjective - what is considered an "incorrect" build by one player, may be "optimal" for another. If you want to make the game harder, focus your skill points and perks on non-combat skills: Speech, Pickpocket, Lockpicking, etc. You can also put skill points and perks on combat skills that you won't be using in combat. This will level up your character (which will make enemy level difficulty scale to your level), while at the same time, not make your character stronger for combat.

With the Dragonborn DLC installed, perks may be undone and redistributed at the cost of one dragon soul per skill tree.

At the end of the At the Summit of Apocrypha quest, you will have access to different portals (one for each skill) which allows you to clear and regain any perks in that skill tree, at the cost of one dragon soul. You remove all perks from a single skill perk tree and can use these reclaimed perks on unlocking any perks you wish, including perks taken from said perk tree. By reading the Black Book, Waking Dreams, you can return to Apocrypha and alter the skill trees whenever you wish.

Are there any hard quests that I've completely missed, or do you think that now if I start new questlines with my increased stats, the game would level up the difficulty for me?

It is subjective to state which quests are "Hard quests". Just have a look here for a list of all quests and see which ones appear to be hard for you. As for the second part of your question, the unmodded game only checks your current level and the difficulty settings in the options to compute how difficult the enemies will be for you.

Excerpts from UESP wiki's "Leveling - Effects of Leveling" article:

Various aspects of the game are leveled. This means that as your character increases in level, some enemies become more challenging but also the quality of the items you find becomes better. However, the leveling system in Skyrim has been altered from that used in Oblivion, in response to criticisms of Oblivion's leveling system.

Different locations in Skyrim have different inherent difficulties. In other words, some dungeons are designed to be too difficult for low-level characters to enter. More challenging dungeons are generally located at higher elevations, meaning that early in the game players may want to avoid mountainous regions. However, more difficult dungeons contain better rewards. In addition, some high-quality items can be randomly found even early in the game.

... Bandit NPCs are always a fixed level for their name (Bandits are level 1, Bandit Thugs are level 9, Bandit Highwaymen are level 14, etc). The player's level affects the range of possible bandit types generated within a bandit dungeon, and probably the frequency, but does not seem to affect the resulting stats except in a few rare cases. Lower variant bandits remain reasonably common even when more dangerous bandits are available.

v1.9 Patch - 'Legendary' difficulty

Patch 1.9 adds a sixth difficulty level: Legendary. It reduces damage dealt by the player to x0.25 and increases damage taken by the player to x3.

Mods

If playing on the PC, you could also make the game harder with mods:

The mod, Pluto's Improved Skyrim Experience (PISE) has a "More Intense Level Scaling" component. It makes enemies more stronger relative to your level, compared to the vanilla enemy scaling. PISE also features harder sneaking, more enemy spawns, harder enemies and an overhauled enemy AI. The mod, Path of Shadows, a major stealth overhaul mod, also makes sneaking harder.

Other mods that may make the game harder:

DFB - Random Encounters - adds different random encounters from vanilla: Vampires, Falmers, Werewolves, Dwarven Spiders, Spheres, Centurion, etc.
High Level Enemies - has a feature that allows certain or all enemies to scale with your level
Deadly Dragons - overhauls dragons to make them more challenging to fight
Balanced Magic - designed to 'balance' the game's magic spells - make the spells do damage and consume mana appropriate to your magic perks, and lessens the effect of abusive perks like the 100% stagger change of the 'Impact' perk. It also affects enemy mage NPCs, making them more challenging
Auto-cast Racial Powers Plugin - "Auto-cast racial powers will activate for NPCs, both enemy and friendly, making fights much more interesting and adding a new element to prioritizing targets in larger fights!"

Creature mods that increase creature spawn points, number of spawns, and also improves creature AI:
(I recommend using only one creature mod, to avoid conflicts and other issues.)

Real Wildlife - Skyrim
Realistic Animals and Predators
Skyrim Creatures Alive

Solution 2:

I had this same problem. This is what I am trying. YMMV.

You can try this play style. Go without killing anyone for as many levels as possible on Adept or Master difficulty. This was proposed by @Alex as well in the comments to your question.

It is a very challenging play style, for quests where I simply have to kill as part of the quest objective, it gets difficult, but even then with Mayhem/Frenzy/Fear etc it is possible that your quarry gets killed.

For an example of this challenging play style check out Felix The Peaceful Monk

Solution 3:

I had the same issue! Here's how to make it difficult:

No glitch use
Leveling up skills using cheap ways is banned (like leveling up restoration by jumping on blacksmith fire and heal)
You can only put perks in four trees only (example: one hand + destruction + light armor + archery)
(this is most important). You can't pick perks that require skill level 70 or more! Because such perks are overpowered, like dual savegery and elemental protection
enchanting can accept only 2 perks, because more will make you overpowered
Your character is allowed to use 2 shouts only. You assign them at the begining of the game and don't use other shouts execept for quests