iphone x and lens for camera - should I be worried about scratches

Most surface scratches on any camera lens don’t matter a huge amount due to how light focuses. That being said, you’ll want to not scuff too badly or crack the housing. Apple has had a long time to get the hardness right on the glass and cameras, so I would expect you’d have a low chance of issues - especially if you have a case.

Drops to the body (to the ground from a table / case or not) and internal failure are much more likely to compromise image quality than a physical failure of the lens.

Also note - even without a case, the physica of the metal band surrounding the camera make it very hard to hit the part of the glass where the sensor / light enter. Nothing is going to stop an ice pick or diamond ring that targets “ground zero” on the cover, but most unintentional drops and scrapes should be away from the light path by design.