Can I say that the 'to-infinitive which is part of predicative is an object?
Solution 1:
[1] He asked me [to come in].
[2] Jane's house was not easy [to find].
No, the bracketed infinitival clauses are not objects.
[1] is a catenative construction where the infinitival clause to come in is catenative complement of asked. Me is the direct object of asked and the understood subject of to come in.
In [2] the infinitival clause to find is complement of the adjective easy -- they form the adjective phrase easy to find.