Pytorch - Inferring linear layer in_features

I am building a toy model to take in some images and give me a classification. My model looks like:

conv2d -> pool -> conv2d -> linear -> linear.

My issue is that when we create the model, we have to calculate the size of the first linear layer in_features based on the size of the input image. If we get new images of different sizes, we have to recalculate in_features for our linear layer. Why do we have to do this? Can't it just be inferred?

Why do you expect the linear layer to infer its input size? What if you intentionally want to change this size (i.e., the conv layer output channels or whatever). I believe your work should be parameterized (i.e., controlled by well-defined parameters). A workaround is to always transform the input images into a defined shape. But, I do not recommend so since this may heavily affect your model's accuracy because transforming includes losing pixels (i.e., the features of your input).

As of 1.8, PyTorch now has LazyLinear which infers the input dimension:

A torch.nn.Linear module where in_features is inferred.