Is it secure to use self signed certificate on FTP but not in HTTPS? Why?
I'm trying to generate self signed certificates (SSC) for most of the local services. Cryptography has been a bit complicated to understand, mostly the PKI.
Is there any conceptual reason why we don't use SSCs on HTTPS, but seems to be fine (may be wrong here) for FTP secure?
This is the way I'm generating any of them, instead of a certificate signing request:
openssl req -x509 -nodes -days 365 -sha256 -newkey rsa:4096 -keyout server.pkey -out server.cert
The assumption in other answers here is that HTTPS is only used for browsing the web by general consumers. Granted, that a huge majority of HTTPS traffic is the web, but the two are only protocols, and there are edge cases (e.g. a REST based API) where HTTPS could be as specialist as FTP, in which case the argument for and against self-signed certs don't differ.
A service where two end-points communicate over HTTPS and it's unlikely to expand, could be, and often is, operated with self-signed. There is no difference from the point of view of the protocols, only the use case.
PKI signed certificates come into their own when there's a one-to-many relationship between the end-entity (server) and relying parties (clients). Self-signed certificates in this scenario would be cumbersome as clients would individually need to review and trust each server certificate (ignoring things like Windows Group Policy for trust-anchor distribution). If you extrapolate this to multiple servers, such as the WWW, then the challenge gets even larger. Add into the mix the fact that most users of the WWW aren't tech-savvy then self-signed become a burden.
On the other hand, setting up a full PKI for two machines to communicate is overkill, given the overhead of running a secure PKI service. If all you're doing is sending a file between two servers, or sending a REST API message between two machines, then self-signed certificates could be considered acceptable here. During the configuration of the two end-points, most probably by tech-savvy admins, they could review the certificate and install it.
A compromise is to use commercial CA, where the work of operating a trusted CA is done for you. There's no need to operate a PKI and certificates are trusted by many potential users. The only additional workload is that the operator of the server will need to enrol for a certificate from the CA (possibly for a fee) and install that on their server instead of generating a self-signed. This assumes that your machines are connected to the Internet though.
On a similar note to the previous paragraph, your organisation may operate their own internal PKI, and clients already have the root CA certificate in their trust-anchor store. The discussion for and against self-signed vs CA signed become fuzzier now as there's little extra effort involved in using a CA signed certificate.
The bottom line is: HTTPS or FTPS as protocols don't care whether you use CA signed or self-signed certificates. You need to analyse your requirements on a case-by-case basis and decide what model to use.
Is there any conceptual reason why we don't use SSCs on HTTPS, but seems to be fine (may be wrong here) for FTP secure?
I think you are wrong here, i.e. it is not acceptable.
But, FTP is a dying technology as for example shown by the major browsers removing support for it. There only comparably few FTP servers in the first place, at least compared to the number of web sites. Even less of these support FTPS (which is not the same as SFTP), i.e. FTP with TLS. And the major browser never supported FTPS in the first place.
So what you have is essentially the junk yard of the internet, mostly abandoned technology which is only kept for some special cases. General public access is usually not one of these any more since it got replaced by HTTP and HTTPS. So one can say that nobody cares a lot what happens there.
Still, it is not acceptable. In times where certificates are easy to get and free (Let's Encrypt and others) it is not that much of an effort any more to get a proper certificate instead of a self-signed.