Regarding silent installation using Setup.exe generated using Installshield 2013 (.issuite) project file

I have a setup.exe file generated using installshield suite project (.issuite) file. For an admin or a general user with sufficient privileges, one of our IT admin is able to do a silent installation using -

setup.exe /silent

However, when tried to push the installs using SCCM (System Center Configuration Manager) or using PSEXEC (both use System account to install), the install fails. The install exist with an exit code 3.

I did some research around this and found out that system account needs a setup.iss (answer/response) file to write to / read from. Did not quite understand what it means.

I also tried using -

setup.exe /r

but then there is no setup.iss generated and install fails with a popup message as -

The setup command line is invalid. The setup cannot proceed.

Is it also true that setup.iss file cannot be created if we are using custom dialogs / wizard steps?

Please let me know how may I silently install using system account credentials.

Short Answer

UPDATE: is this your own suite? If so, you may be able to improve your silent running configuration specified in the suite settings to allow each embedded setup to run correctly in silent mode.

The switch you are using to install silently is correct, but instead of trying to run the whole suite silently, I would try to:

Extract the embedded setups and components from the suite project EXE file to a disk location (details below).
Use your deployment system - which in your case is SCCM - to install the required components in your own order whilst eliminating any unnecessary runtimes that are already on your SOE - for example Visual C++ Runtime, etc...

In order to extract the components from a suite setup.exe (or equivalent meaningful name such as InstallShield2018Premier.exe) run this command line:

setup.exe /stage_only

You have to specify an output location, and the extracted contents from your suite EXE will be put there, and crucially it will also download any components set to be downloaded off the Internet to the staging folder.

The feature to download components from the Internet is not compatible with corporate deployment via deployment system such as SCCM - any attempts to access the Internet will fail for an EXE run via SCCM.

This Internet download issue could very well be the problem you experienced, but there is no way to tell without a proper log file. Did you create a proper log file by running the suite with the /log switch?

See the full list of available command line switches for suite projects here: Advanced UI and Suite/Advanced UI Setup.exe Command-Line Parameters.

The embedded setups - now extracted - could be of various types - each with its own way of installing silently depending on what technology they were made with. Details below.

Further Details

The below became way too long - I believe the above information should suffice to get the job done in your case, but I will leave the content below in case it is helpful. Maybe it will be useful later for others:

What is a setup.exe?

Just for the record: setup.exe files can be a lot of different things. There are some observations about it here: How can I use powershell to run through an installer? It can be an Installshield setup or suite, an Advanced Installer setup or suite, a proprietary setup file from "any vendor", a self-extracting archive (maybe zip based), a WiX Burn bundle, an Inno setup - or any number of other types of executable usually created by some sort of deployment tool, but they can also be entirely proprietary and essentially just be regular Windows portable executable files.

Installshield Suite EXE Files

Installshield Suite Projects are capable of installing several installers - of various types - in sequence. MSI files, legacy installers, custom runtime installers, etc...
Suites are wrapped in setup.exe files (or an equivalent meaningful EXE name such as InstallShield2018Premier.exe), but they feature a different set of command line switches than normal setup.exe files from Installshield (which contain a single product to install).
The full list of available command line switches for suite projects can be found here: Advanced UI and Suite/Advanced UI Setup.exe Command-Line Parameters.
You can extract the embedded setups from a suite setup.exe AND cache locally whatever the suite project tries to download off the internet by using the command line switch: setup.exe /stage_only
There are similar bootstrapper features in the WiX toolkit and in Advanced Installer, and the resulting setup.exe files they create can also be extracted, but that is another story.

Regular Installshield EXE Files

Apart from suite projects, Installshield-built setup.exe files has a number of additional flavors. An Installshield setup.exe can also be:

A legacy script based installer (non-MSI setups created prior to the development of Windows Installer)
A setup.exe wrapping an Installscript MSI (which is a special form of MSI installer featuring a Win32 dialog GUI - very error prone these setups)
A setup.exe wrapping a Basic MSI (which is a proper MSI installer with standard MSI GUI)
Potentially other flavors that I am not aware of.

These different setup.exe flavors have similar command lines and they differ quite a bit from the command lines for suite files (explained above). I believe there are some variations for the command line for different versions of Installshield, but here is the documentation for the most recent version of Installshield as of now - which is Installshield 2018: Setup.exe and Update.exe Command-Line Parameters.

Silent Install for Installshield setup.exe Files

So now we have many different types of setup.exe files delivered by Installshield. We have the suite project setup.exe - which is really a bundle of setup files to run in sequence - and not a single Installscript setup at all. Then we have legacy Installscript setup.exe files that are not based on Windows Installer, but are old-school installers that are more and more rare to see made with Installshield. And finally there are two flavors of setup.exe file that are Windows installer related: Installscript MSI and Basic MSI. Now, how can you install these different setup.exe files in silent mode?

Installshield suites

A suite setup.exe can be deployed silently using the installation switch you have already tried: setup.exe /silent. But as stated above it may start pulling components down from the Internet. This is unsuitable for corporate deployment. Not at all recommended to try - unless you are a home user or a small office with no deployment solution.

Installscript (non-MSI)

You can run legacy setup.exe files silently by recording a response file with setup.exe /r and then you run the install silently with setup.exe /s. There are some detailed samples for how to do this here.

Installscript MSI

Installscript MSI setup.exe files are handled the same as legacy Installshield setup.exe files. See the above commands and links.

It used to be that you needed to add an /SMS switch when deploying via a deployment system to ensure the setup.exe didn't exit prematurely. According to the 2018 Installshield documentation this switch is no longer needed.

It is possible to extract the embedded MSI file from an Installscript MSI and install it without a setup.exe launcher. This is a little more involved. Some details here.

Basic MSI

Basic MSI setup.exe files contain MSI files that are capable of running silently without a response file. It is generally better to perform an administrative image of such a setup.exe to extract the embedded MSI and deploy that MSI directly instead of using the setup.exe wrapper.

So for Basic MSI setup.exe wrappers you can either 1) extract files from the embedded MSI by running and administrative image on our setup.exe - the command for this is setup.exe /a (here is a more general description of administrative installations) , or you can 2) pass installation parameters to the embedded MSI file by specifying the /v parameter to the setup.exe: Setup.exe /s /v"/qn INSTALLDIR=D:\Destination".

Check the full Setup.exe and Update.exe Command-Line Parameters.

Extracting Files from Setup.exe

Here is an answer with details on how to extract content from various setup.exe files: Programmatically extract contents of InstallShield setup.exe.

Essentially you use setup.exe /stage_only for suite executables. And setup.exe /a for Basic MSI and Installscript MSI executables. And setup.exe /s /extract_all for legacy Installscript executables.

If you have a setup.exe installer that is just not cooperating, try launching it, wait for its first setup dialog to appear, and then look for extracted files in the temp folder.