What is really meant by "Invalid B-tree node size"? How to solve it without expensive 3rd party softwares?

It basically means the file can't be looked in the B-Tree. A more in-depth explanation can be found at https://www.easeus.com/mac-file-recovery/invalid-b-tree-node-size.html

Option 1 - Disk Utility

It's an inbuilt disk repair utility that repairs most commonly seen hard drive problems.

1. Choose Apple menu, click Restart.
2. Press and hold the Command and R keys until the Apple logo appears and then release the keys.
3. Click Disk Utility and click Continue.
4. In the sidebar, select the disk with invalid b-tree node size and you want to repair, click the First Aid button.
5. Click Run.

If Disk Utility can't repair your disk, run it again. Or else, move on the next solutions.

Option 2 - Fsck command

The fsck command is used to repair disk and file system problems on your Mac.

1. Start your Mac in single-user mode. Restart it, and then press and hold the Command+S keys while it boots.
2. Type the command /sbin/fsck -fy into the terminal and press Enter to start a file system check.
3. When the fsck command has your disk checked and repaired, type reboot and press Enter. Your Mac will reboot and return to the usual login screen.

Option 3 - Reformat

Take the Mac hard drive reformatting as the last step, because this formatting process will erase all data on the drive completely. Make sure you've backed up your hard drive with Time Machine.

1. Launch Disk Utility. Click the icon for your external hard drive in the sidebar on the left.
2. Click the Erase tab along the top of the window.
3. From the Volume Format menu, choose Mac OS Extended (Journaled).
4. Enter a name for the external hard drive in the Name field.
5. Click the Erase button.

Disclaimer This answer is originally from https://www.easeus.com/mac-file-recovery/invalid-b-tree-node-size.html and was helpful for me when i had the same issue.

Whenever you save a file on Mac, the Mac operating system creates a unique entry for the file on the index table with B-tree format. So each time you access a file, the Mac operating system navigates this tree to locate the file. Once B-tree is corrupted, the Mac operating system fails to and you will see the error message: "Invalid B-tree node size. This disk needs to be repaired".

Read more from this post: https://iboysoft.com/mac-data-recovery/invalid-b-tree-node-size.html

I'm afraid that you can only reformat your disk to fix this error. But this operation will erase all data on the hard drive completely. If you have important data, you should use data recovery software to restore them before reformatting.

Is there any solution to this problem without using an expensive 3rd party software? Has anyone fixed this problem before?

The usual solution to disk problems is to maintain a backup of your files, so that when things go wrong, you have another copy of your files. Apple has included a 'fixer': Time Machine, which is an excellent and effortless backup utility.

The alternative is data recovery software, and yes you'll need to buy that. Data Rescue 6 offers recovery from as little as $19, and you can see what it will recovery before buying.

https://www.prosofteng.com