Can't give unusual gem to appraiser

Every time I try to bring an unusual gem to an appraiser they say "Wanna talk? finish those jobs for Brynjolf first" but I don't know what I have to do for Brynjolf cause I have so many quests to do. Anyone know what quest it is? Is there anything else I have to do to bring an unusual gem to an appraiser? I can't get rid of the unusual gem.

The quest is "Bring the Unusual Gem to an appraiser", which appears when you find an unusual gem. You are directed to Riften, and when asking about that gem all around:

Eventually, the Dragonborn will be directed to speak with Vex, a member of the Thieves' Guild who reveals that the unusual gem is in fact a Stone of Barenziah, and that there are 24 of them in total.

Source

That unlocks a new "quest", in which you have to find all gems, the Stones of Barenziah.

Spoiler alert!

More info on that wiki: Stones of Barenziah. Here is the complete list of stones locations.

1. Ansilvund, in Eastmarch, contains a stone near where you encounter Fjori inside the burial chambers.
2. Black-Briar Lodge, outside Riften, the master bedroom.
3. College of Winterhold on the shelves of the Arch-Mage's Quarters.
4. Dark Brotherhood Sanctuary, look in Astrid's room for the stone. This stone can only be found while completing the “With Friends Like These...” or “Destroy the Dark Brotherhood!” quests.
5. Dead Crone Rock, south-west of Markarth. The stone can be found on an altar near a Word Wall at the top.
6. Fellglow Keep, can be found east-north-east of Whiterun. Look on the counters of the workroom to find this stone.
7. Hob's Fall Cave. The stone is inside the Necromancer's sleeping area.
8. Markarth, the Treasury House, search the master bedroom to find the stone.
9. Markarth, in the Dwemer Museum. The stone will be inside, on a table behind a locked door.
10. Pinewatch, a farm north-east of Falkreath. Inside the farmhouse, look for a secret entrance into an underground cave. Inside the Bandit Sanctuary, look for a locked door. Lockpick the door, inside you'll find the stone on a bookshelf.
11. Rannveig's Fast, north-east of Rorikstead and south-east of Morthal. Look for a prison area, and search the table with the Alchemy Lab for the stone.
12. Riften, Mistveil Keep, the Jarl's Quarters.
13. Solitude, just outside town, on The Dainty Sload, a boat. Look in the Captain's Quarters, on the table.
14. Solitude, in Proudspire Manor. You'll find the stone in the master bedroom.
15. Solitude, in the Blue Palace. Look in the Jarl's quarters, on one of her shelves.
16. Stony Creek Cave, can be found north of Ansilvund, south-east of Windhelm. Look in the area where you encounter the Bandit Wizard.
17. Sunderstone Gorge south of Gjukar's Monument and north of Moss Mother Cavern, in Falkreath. You'll find a Word Wall inside, with the added bonus of a nearby stone.
18. Thalmor Embassy, look in Elenwen's Solar while completing the "Diplomatic Immunity" quest for Delphine in Riverwood. If the stone is not in the Solar, it will likely be in the Reeking Cave on the way out.
19. Whiterun, in Dragonsreach, explore the Jarl's bedroom to find a stone.
20. Whiterun, Kodlak's room in Jorrvaskr.
21. Whiterun, Hall of the Dead, check the wall crypts to find another stone.
22. Windhelm, House of Clan Shatter-Shield, simply search the first upstairs bedroom to your left to find the stone.
23. Windhelm, Palace of the Kings, look on the table in Wuunferth the Unliving's Quarters.
24. Yngvild, a cavenorth of Winterhold and east of Dawnstar. Search behind the throne to find a chamber containing the stone.