How can I get... just a little bit higher?

You can use many characters attributes and skills to acquire these:

• Wind Catcher is used to place a wind current anywhere;

• if you have the Traveler as Geo, their elemental skill can create a platform to climb;

• if you have Venti, his elemental skill also creates a wind current(but only after 1st Ascension);

• if you have Albedo, his elemental skill spawns elevator like platform;

• if you have Keqing, her elemental skill could be used as a short range teleport;

• if you have Zhongli, his elemental skill summons a pillar and you can jump on top of it;

• if you have Kazuha, his elemental skill includes a high jump;

• if you have Xiao, his elemental burst enables high jumps and dashing above the ground.

Also, characters height differs, sometimes it could be enough to switch to higher one. For example, f2p Kaeya is one of the tallest.