Is there any way of improving Doom graphics when running it on DOSBox?

If you don’t like the blocky, rasterized graphics you are seeing when attempting to play Doom in DOSBox, the better alternative is to use a native OS interpreter of the Doom engine such as GZDoom; a newer and maintained GNU licnesed version of ZDoom. These modern Doom engines use the vector graphics capabilities of modern systems. As they describe on their websites:

“ZDoom is a family of enhanced ports of the Doom engine for running on modern operating systems. It runs on Windows, Linux, and OS X, and adds new features not found in the games as originally published by id Software.”

“Older ZDoom ports may be used and distributed free of charge. No profit may be made from the sale of it. GZDoom and its descendants from version 3.0.0 on are licensed GPL and are subject to the terms and restrictions of the new license.”

Installing GZDoom is fairly easy and explained clearly on the ZDoom Wiki. You just need to be sure to have the WAD files you need to play. They even have a very good overview of all of the WAD files you can use with ZDoom.

“An IWAD - short for Internal WAD - is the main resource file for a Doom-engine game, containing all the game's original sounds, levels, and graphics. ZDoom needs an IWAD file in order to play; without one the ZDoom engine has no game data to use.”

If you have a downloaded copy of Doom, the WAD files should be in there. Get them out of that DOSBox setup, move them to GZDoom on your main OS and you should be in business.

FWIW, after writing this answer I went and downloaded GZDoom, found a WAD for the demo version of Doom and I was in business. Screenshot below. Note that the graphic textures are still pixelated, but the edges of surfaces are now smooth like a vector rendered 3D game. Not perfect, but an improvement over 100% rasterized 3D graphics.

Yes you're becoming too greedy. GZDoom is as good as it's going to get, because those textures and sprites were made as you said nearly 30 years ago. They were not made to take advantage of future graphics features, they were a very very small company back then. The only other thing you could look for us a high res textures pwad file to supplement them. But the engine and effects is still going to be the same experience.

I dont have experience with ZDoom/GZDoom, but I have used Doomsday engine which works much in the same way: You drop in your WAD, and you are good to go. With Doomsday you can play with the vanilla textures and sprites or you can add community-developed resource packs to improve sound, music, textures and change monsters from 2D sprites to 3D models.