Red Hat Linux tuning for a single application with a few TCP clients [closed]

Performance tuning is as much a black art as it is a science. Complete books have been written about it, include quite a bit as part of the vendor documentation

You start with good monitoring, add load, get a baseline and only then you start adjusting and observing if your adjustment did or did not have the intended result.
Rinse and repeat until you get the optimal result.

You don't mention your major RHEL or CentOS release but they all include automatic tuning as well, with for instance the tuned daemon. Predefined performance profiles can be selected from tuned-adm list and then activated with tune-adm profile <profile-name>.

For your use-case the RHEL 7 profile latency-performance sounds like the one you might benefit from:

latency-performance A server profile focused on lowering latency. This profile is recommended for latency-sensitive workloads that benefit from c-state tuning and the increased TLB efficiency of transparent huge pages. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100. It enables transparent huge pages, uses cpupower to set the performance cpufreq governor, and requests a cpu_dma_latency value of 1.

For more general Linux tuning Brandan Gregg has made a number of visual diagrams of which tool to use on what part of your system that make really nice cheat sheets:

Tools:

Which options to use in sar to monitor:

And closely related:

Translate observations into tuning with: