Can I give a custom-enchanted dwarven item to Calcelmo for his quest?

Calcelmo requires an item that is the same as the one he mentions in the letter (it doesn't have to be the same item, though). Player-enchanted items are not the same as items found in the world, so it won't work, you'll need to find another Dwarven Sword of Devouring to give to Calcelmo. If he asked you for a non-enchanted item, smithing it will work.

If you're playing on PC, you can use the console command help "dwarven sword of devouring" to find the baseid (8 digit code) of that item, then use player.additem baseid 1 to give yourself a copy of that sword.