Any downsides to switching "home xbox" setting with my brother?

My brother recently bought an Xbox One.

I have a lot more games on my account than he has on his, but I am wondering if the following setup might give either of us a downside?

• I add my account to his Xbox, and set it as my home xbox
• If he at some point buys games I haven't bought, we do the same the other way around, add his account to my xbox and set my xbox as his home xbox

Would this give me a disadvantage on my own xbox? Would I have to sign in more often? With the "Home xbox" setting I gather that he will get access to all my games. Can we play my games at the same time? ie. he doesn't have Battlefield 4, but can both he and I both log on to Battlefield 4 using my game and play it together?

Is this illegal in any way? We're both under the same roof if that matters at all.

Basically my question comes down to this: With the above setup, will either of us run into any problems or disadvantages?

Let me try to clarify.

• Two xboxes under the same roof
• Two accounts, mine, and his
• Proposed setup:
  • I add my account to his Xbox, and set his Xbox as my home Xbox
  • (Optional for now) He adds his account to my Xbox, and sets my Xbox as his home Xbox
  • We can then download and install any games from my account onto his Xbox
• I understand that I can still play my own games on my own Xbox
• I gather that he will also be able to play my games on his Xbox

The questions are:

• Can we do it at the same time? Ie. both fire up my accounts Battlefield 4 (he doesn't have BF4 on his account) and play it, together?
• Will there be any downsides to this? Such as both xboxes (which are now thus not our "home xbox" any more) require more frequent logins, or other such things?

One of the reasons I ask is that it seems this is some kind of loophole. I mean, with the above setup, if this is both legal and have no downsides (or at least none worth complaining about), wouldn't this effectively mean that we only have to buy 1 copy of a game from now on, and still be able to both play it?

Can we do it at the same time? Ie. both fire up my accounts Battlefield 4 (he doesn't have BF4 on his account) and play it, together?

Yes, this is possible. I did it with Destiny, CoD Ghosts and some other games.
And your brother would also get xbox live gold access if you have it.

Is this illegal in any way?

Will there be any downsides to this? Such as both xboxes (which are now thus not our "home xbox" any more) require more frequent logins, or other such things?

I am using this for about one month and have no problems at all. As far as I know is this intended to used like this.
You just need to login once to setup the home xbox. After the home xbox is set your brother can login as usual.