List of safe to remove applications

Ubuntu comes with a lot of GNOME applications pre-installed for giving the users an overall good desktop experience. Many of them have a deep system integration ... so it's not recommended to remove them - a good example is nautilus. But there are features that are quite safe to remove ... such as the controversially discussed online search integration that Canonical provides in Ubuntu.
As this is an aspect being more opinion based ... I give you a list of packages that I have removed :

account-plugin-aim  
account-plugin-facebook  
account-plugin-flickr  
account-plugin-jabber  
account-plugin-salut  
account-plugin-yahoo  
aisleriot  
gnome-mahjongg  
gnome-mines  
gnome-sudoku  
landscape-client-ui-install  
unity-lens-music  
unity-lens-photos  
unity-lens-video  
unity-scope-audacious  
unity-scope-chromiumbookmarks  
unity-scope-clementine  
unity-scope-colourlovers  
unity-scope-devhelp  
unity-scope-firefoxbookmarks  
unity-scope-gmusicbrowser  
unity-scope-gourmet  
unity-scope-guayadeque  
unity-scope-musicstores  
unity-scope-musique  
unity-scope-openclipart  
unity-scope-texdoc  
unity-scope-tomboy  
unity-scope-video-remote  
unity-scope-virtualbox  
unity-scope-zotero  
unity-webapps-common

Note :

Uninstalling unity-webapps-common removes the Amazon Search feature from the Unity desktop environment. In Ubuntu 15.10 I had to reinstall unity-webapps-service afterwards.
The reinstallation of the package unity-webapps-service wasn't necessary in Ubuntu 14.04.

You first need to define what considers "break my system". A system without a desktop is not broken as is.

Besides that: the safest way is what you did and to take a look at this by "category" from Ubuntu Software Center and the "installed" group.

Accessories
Books & Magazines
Developer Tools
Education
Fonts
Games
Graphics
Internet
Medicine
Office
Science & Engineering
Sound & Video
Themes & Tweaks
Universal Access

Besides "accessories" any of these should be safe to remove and will not remove anything important since they all point to applications. Remove them 1 by 1 and do pay attention to the list of packages it is offering to delete just in case.

And when in doubt check the packages online for what is depending.

1 more comment: what you did was probably overkill; to get those icons back in system settings all you had to do is re-install that specific packages. This should have been enough to fix it:

sudo apt-get remove unity-control-center
sudo apt-get install unity-control-center