Writing to a closed, local TCP socket not failing
Solution 1:
This is what the Linux man page says about write and EPIPE:
EPIPE fd is connected to a pipe or socket whose reading end is closed.
When this happens the writing process will also receive a SIG-
PIPE signal. (Thus, the write return value is seen only if the
program catches, blocks or ignores this signal.)
When Linux is using a pipe or a socketpair, it can and will check the reading end of the pair, as these two programs would demonstrate:
void test_socketpair () {
int pair[2];
socketpair(PF_LOCAL, SOCK_STREAM, 0, pair);
close(pair[0]);
if (send(pair[1], "a", 1, MSG_NOSIGNAL) < 0) perror("send");
}
void test_pipe () {
int pair[2];
pipe(pair);
close(pair[0]);
signal(SIGPIPE, SIG_IGN);
if (write(pair[1], "a", 1) < 0) perror("send");
signal(SIGPIPE, SIG_DFL);
}
Linux is able to do so, because the kernel has innate knowledge about the other end of the pipe or connected pair. However, when using connect, the state about the socket is maintained by the protocol stack. Your test demonstrates this behavior, but below is a program that does it all in a single thread, similar to the two tests above:
int a_sock = socket(PF_INET, SOCK_STREAM, 0);
const int one = 1;
setsockopt(a_sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
struct sockaddr_in a_sin = {0};
a_sin.sin_port = htons(4321);
a_sin.sin_family = AF_INET;
a_sin.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
bind(a_sock, (struct sockaddr *)&a_sin, sizeof(a_sin));
listen(a_sock, 1);
int c_sock = socket(PF_INET, SOCK_STREAM, 0);
fcntl(c_sock, F_SETFL, fcntl(c_sock, F_GETFL, 0)|O_NONBLOCK);
connect(c_sock, (struct sockaddr *)&a_sin, sizeof(a_sin));
fcntl(c_sock, F_SETFL, fcntl(c_sock, F_GETFL, 0)&~O_NONBLOCK);
struct sockaddr_in s_sin = {0};
socklen_t s_sinlen = sizeof(s_sin);
int s_sock = accept(a_sock, (struct sockaddr *)&s_sin, &s_sinlen);
struct pollfd c_pfd = { c_sock, POLLOUT, 0 };
if (poll(&c_pfd, 1, -1) != 1) perror("poll");
int erropt = -1;
socklen_t errlen = sizeof(erropt);
getsockopt(c_sock, SOL_SOCKET, SO_ERROR, &erropt, &errlen);
if (erropt != 0) { errno = erropt; perror("connect"); }
puts("P|Recv-Q|Send-Q|Local Address|Foreign Address|State|");
char cmd[256];
snprintf(cmd, sizeof(cmd), "netstat -tn | grep ':%hu ' | sed 's/ */|/g'",
ntohs(s_sin.sin_port));
puts("before close on client"); system(cmd);
close(c_sock);
puts("after close on client"); system(cmd);
if (send(s_sock, "a", 1, MSG_NOSIGNAL) < 0) perror("send");
puts("after send on server"); system(cmd);
puts("end of test");
sleep(5);
If you run the above program, you will get output similar to this:
P|Recv-Q|Send-Q|Local Address|Foreign Address|State|
before close on client
tcp|0|0|127.0.0.1:35790|127.0.0.1:4321|ESTABLISHED|
tcp|0|0|127.0.0.1:4321|127.0.0.1:35790|ESTABLISHED|
after close on client
tcp|0|0|127.0.0.1:35790|127.0.0.1:4321|FIN_WAIT2|
tcp|1|0|127.0.0.1:4321|127.0.0.1:35790|CLOSE_WAIT|
after send on server
end of test
This shows it took one write for the sockets to transition to the CLOSED states. To find out why this occurred, a TCP dump of the transaction can be useful:
16:45:28 127.0.0.1 > 127.0.0.1
.809578 IP .35790 > .4321: S 1062313174:1062313174(0) win 32792 <mss 16396,sackOK,timestamp 3915671437 0,nop,wscale 7>
.809715 IP .4321 > .35790: S 1068622806:1068622806(0) ack 1062313175 win 32768 <mss 16396,sackOK,timestamp 3915671437 3915671437,nop,wscale 7>
.809583 IP .35790 > .4321: . ack 1 win 257 <nop,nop,timestamp 3915671437 3915671437>
.840364 IP .35790 > .4321: F 1:1(0) ack 1 win 257 <nop,nop,timestamp 3915671468 3915671437>
.841170 IP .4321 > .35790: . ack 2 win 256 <nop,nop,timestamp 3915671469 3915671468>
.865792 IP .4321 > .35790: P 1:2(1) ack 2 win 256 <nop,nop,timestamp 3915671493 3915671468>
.865809 IP .35790 > .4321: R 1062313176:1062313176(0) win 0
The first three lines represent the 3-way handshake. The fourth line is the FIN packet the client sends to the server, and the fifth line is the ACK from the server, acknowledging receipt. The sixth line is the server trying to send 1 byte of data to the client with the PUSH flag set. The final line is the client RESET packet, which causes the TCP state for the connection to be freed, and is why the third netstat command did not result in any output in the test above.
So, the server doesn't know the client will reset the connection until after it tries to send some data to it. The reason for the reset is because the client called close, instead of something else.
The server cannot know for certain what system call the client has actually issued, it can only follow the TCP state. For example, we could replace the close call with a call to shutdown instead.
//close(c_sock);
shutdown(c_sock, SHUT_WR);
The difference between shutdown and close is that shutdown only governs the state of the connection, while close also governs the state of the file descriptor that represents the socket. A shutdown will not close a socket.
The output will be different with the shutdown change:
P|Recv-Q|Send-Q|Local Address|Foreign Address|State|
before close on client
tcp|0|0|127.0.0.1:4321|127.0.0.1:56355|ESTABLISHED|
tcp|0|0|127.0.0.1:56355|127.0.0.1:4321|ESTABLISHED|
after close on client
tcp|1|0|127.0.0.1:4321|127.0.0.1:56355|CLOSE_WAIT|
tcp|0|0|127.0.0.1:56355|127.0.0.1:4321|FIN_WAIT2|
after send on server
tcp|1|0|127.0.0.1:4321|127.0.0.1:56355|CLOSE_WAIT|
tcp|1|0|127.0.0.1:56355|127.0.0.1:4321|FIN_WAIT2|
end of test
The TCP dump will show also show something different:
17:09:18 127.0.0.1 > 127.0.0.1
.722520 IP .56355 > .4321: S 2558095134:2558095134(0) win 32792 <mss 16396,sackOK,timestamp 3917101399 0,nop,wscale 7>
.722594 IP .4321 > .56355: S 2563862019:2563862019(0) ack 2558095135 win 32768 <mss 16396,sackOK,timestamp 3917101399 3917101399,nop,wscale 7>
.722615 IP .56355 > .4321: . ack 1 win 257 <nop,nop,timestamp 3917101399 3917101399>
.748838 IP .56355 > .4321: F 1:1(0) ack 1 win 257 <nop,nop,timestamp 3917101425 3917101399>
.748956 IP .4321 > .56355: . ack 2 win 256 <nop,nop,timestamp 3917101426 3917101425>
.764894 IP .4321 > .56355: P 1:2(1) ack 2 win 256 <nop,nop,timestamp 3917101442 3917101425>
.764903 IP .56355 > .4321: . ack 2 win 257 <nop,nop,timestamp 3917101442 3917101442>
17:09:23
.786921 IP .56355 > .4321: R 2:2(0) ack 2 win 257 <nop,nop,timestamp 3917106464 3917101442>
Notice the reset at the end comes 5 seconds after the last ACK packet. This reset is due to the program shutting down without properly closing the sockets. It is the ACK packet from the client to the server before the reset that is different than before. This is the indication that the client did not use close. In TCP, the FIN indication is really an indication that there is no more data to be sent. But since a TCP connection is bi-directional, the server that receives the FIN assumes the client can still receive data. In the case above, the client in fact does accept the data.
Whether the client uses close or SHUT_WR to issue a FIN, in either case you can detect the arrival of the FIN by polling on the server socket for a readable event. If after calling read the result is 0, then you know the FIN has arrived, and you can do what you wish with that information.
struct pollfd s_pfd = { s_sock, POLLIN|POLLOUT, 0 };
if (poll(&s_pfd, 1, -1) != 1) perror("poll");
if (s_pfd.revents|POLLIN) {
char c;
int r;
while ((r = recv(s_sock, &c, 1, MSG_DONTWAIT)) == 1) {}
if (r == 0) { /*...FIN received...*/ }
else if (errno == EAGAIN) { /*...no more data to read for now...*/ }
else { /*...some other error...*/ perror("recv"); }
}
Now, it is trivially true that if the server issues SHUT_WR with shutdown before it tries to do a write, it will in fact get the EPIPE error.
shutdown(s_sock, SHUT_WR);
if (send(s_sock, "a", 1, MSG_NOSIGNAL) < 0) perror("send");
If, instead, you want the client to indicate an immediate reset to the server, you can force that to happen on most TCP stacks by enabling the linger option, with a linger timeout of 0 prior to calling close.
struct linger lo = { 1, 0 };
setsockopt(c_sock, SOL_SOCKET, SO_LINGER, &lo, sizeof(lo));
close(c_sock);
With the above change, the output of the program becomes:
P|Recv-Q|Send-Q|Local Address|Foreign Address|State|
before close on client
tcp|0|0|127.0.0.1:35043|127.0.0.1:4321|ESTABLISHED|
tcp|0|0|127.0.0.1:4321|127.0.0.1:35043|ESTABLISHED|
after close on client
send: Connection reset by peer
after send on server
end of test
The send gets an immediate error in this case, but it is not EPIPE, it is ECONNRESET. The TCP dump reflects this as well:
17:44:21 127.0.0.1 > 127.0.0.1
.662163 IP .35043 > .4321: S 498617888:498617888(0) win 32792 <mss 16396,sackOK,timestamp 3919204411 0,nop,wscale 7>
.662176 IP .4321 > .35043: S 497680435:497680435(0) ack 498617889 win 32768 <mss 16396,sackOK,timestamp 3919204411 3919204411,nop,wscale 7>
.662184 IP .35043 > .4321: . ack 1 win 257 <nop,nop,timestamp 3919204411 3919204411>
.691207 IP .35043 > .4321: R 1:1(0) ack 1 win 257 <nop,nop,timestamp 3919204440 3919204411>
The RESET packet comes right after the 3-way handshake completes. However, using this option has its dangers. If the other end has unread data in the socket buffer when the RESET arrives, that data will be purged, causing the data to be lost. Forcing a RESET to be sent is usually used in request/response style protocols. The sender of the request can know there can be no data lost when it receives the entire response to its request. Then, it is safe for the request sender to force a RESET to be sent on the connection.
Solution 2:
You have two sockets - one for the client and another for the server. Now your client is doing the active close.This means TCP's conection termination has been started by the client ( A tcp FIN segment has been sent from the client send).
At this stage you see the client socket in FIN_WAIT1 state. Now what is the state of the server socket now? It is in CLOSE_WAIT state.So the server socket is not closed.
The FIN from the server has not been sent yet. (Why - since the application has not closed the socket). At this stage you are writing over the server socket so you are not getting an error.
Now if you want to see the error just write close(client_fd) before writing over the socket.
close(client_fd);
printf( "Write result: %d\n", write( client_fd, "123", 3 ) );
Here the server socket is no more in CLOSE_WAIT state so you can see return value of write is -ve to indicate the error. I hope this clarifies.
Solution 3:
After having called write() one (first) time (as coded in your example) after the client close()ed the socket, you'll be getting the expected EPIPE and SIGPIPE on any successive call to write().
Just try adding another write() to provoke the error:
...
printf( "Errno before: %s\n", strerror( errno ) );
printf( "Write result: %d\n", write( client_fd, "123", 3 ) );
printf( "Errno after: %s\n", strerror( errno ) );
printf( "Errno before: %s\n", strerror( errno ) );
printf( "Write result: %d\n", write( client_fd, "A", 1 ) );
printf( "Errno after: %s\n", strerror( errno ) );
...
The output will be:
Accepting
Server sleeping
Client closing its fd... Client exiting.
Errno before: Success
Write result: 3
Errno after: Success
Errno before: Success
Client status is 0, server status is 13
The output of the last two printf()s is missing as the process terminates due to SIGPIPE being raised by the second call to write(). To avoid the termination of the process, you might like to make the process ignore SIGPIPE.