Why is strtok changing its input like this?

c strtok

Ok, so I understand that strtok modifies its input argument, but in this case, it's collapsing down the input string into only the first token. Why is this happening, and what can I do to fix it? (Please note, I'm not talking about the variable "temp", which should be the first token, but rather the variable "input", which after one call to strtok becomes "this")

#include <string.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char* argv[]) {
   char input[]="this is a test of the tokenizor seven";
   char * temp;
   temp=strtok(input," ");
   printf("input: %s\n", input); //input is now just "this"
}

When strtok() finds a token, it changes the character immediately after the token into a \0, and then returns a pointer to the token. The next time you call it with a NULL argument, it starts looking after the separators that terminated the first token -- i.e., after the \0, and possibly further along.

Now, the original pointer to the beginning of the string still points to the beginning of the string, but the first token is now \0-terminated -- i.e., printf() thinks the end of the token is the end of the string. The rest of the data is still there, but that \0 stops printf() from showing it. If you used a for-loop to walk over the original input string up to the original number of characters, you'd find the data is all still there.

You should printout the token that you receive from strtok and not worry about the input array because NULLs will be inserted by strtok. You need repeated calls to get all of the tokens:

#include <string.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char* argv[]) {
  char input[]="this is a test of the tokenizor seven";
  char * temp;
  temp=strtok(input," ");
  while( temp != NULL ) {
    printf("temp is \"%s\"\n", temp );
    temp = strtok( NULL, " ");
  }
}

It's because strtok inserts nulls into each separator, which is why you use repeated calls to strtok to get each token. The input string cannot be used once you start using strtok. You don't "fix" it -- this is how it works.

Why is strtok changing its input like this?

Related

Recent Posts