Changing SSH Port: Should I modify only sshd_config, or also ssh_config?
The sshd_config is the ssh daemon (or ssh server process) configuration file. As you've already stated, this is the file you'll need to modify to change the server port.
Whereas, the ssh_config file is the ssh client configuration file. The client configuration file only has bearing on when you use the ssh command to connect to another ssh host. So, in this case, you don't need to modify it. It will be other client machines connecting to your server.