Can hacked 3DS game files be used for Streetpass or online multiplayer?

It depends on the game and how much the developers have put effort in preventing it. For example, Pokemon has very strict restrictions when it comes to the Rated Battles (all Pokemon must be legitimate in terms of statistics and obtainment), but has next to none when it comes to random matchups and local multiplayer.

The problem lies in defining "hacked data" from the end of a game. For a game like Super Smash Bros this is relatively easy, because hacked stats are relatively easy to spot and prevent. On the other hand, there are games like Pokemon that can have teams that from a human viewpoint are obviously illegal (six shiny, 31 IV, teams for example) cannot be defined as illegal from a computer standpoint (because it's not IMPOSSIBLE to get those teams, it's just very, very unlikely). This makes it very hard for any system to be flawless.

As for Streetpass, this is again entirely dependent on the software. Animal Crossing: New Leaf for example offers no protection or checks against hacked streetpass houses.

Developers can protect against importing these files in the first place by creating checksums, but this is entirely up to the developer and these checksums can be broken.