Does having a Steel or Silver sword drawn affect sign damage?
The main diffence between steel and silver sword is that the silver sword deals extra physical damage on monsters.
Against monster: silver sword basic damage + monster bonus damage > steel sword basic damage
Against humanoids: steel basic damage > silver basic damage (no monster bonus damage)
The effect/power of the signs are unaffected by the weapons you are currently holding in your hands. You'll get it's passive boosts just by equipping it.
The signs can be enhanced by leveling up or some specific equipment that has +sign.