XPath to extract all text between two 'p' elements scrapy

Solution 1:

scrapy shell

In [1]: html = """<html>
   ...: <body>
   ...: <p class="pre-wrap ng-binding"
   ...: ng-bind-html="object._source.startup.general_information.project_public_description"
   ...: ng-click="listView.showDetail(object)" role="button" tabindex="0">
   ...:   <div>With the vision of providing creative sustainable solutions for global food crisis,
   ...:     AquiNovo develops innovative, non-GMO, non-hormonal, peptide-based feed additives,
   ...:     addressing the ever-growing demand for fish protein. Company’s additives improve both growth
   ...:     performance and feed utilization, enabling the <strong><em>growth of more fish with less
   ...:             feed</em></strong>. A unique peptide production system, enables large commercial
   ...:     scale production at significant lower cost and carbon footprint. Growing more fish with less
   ...:     feed also promote several SDG’s including the reduction of pressure on fish population in
   ...:     the sea, providing food security and reducing hunger and poverty, climate change and
   ...:     responsible production.&nbsp;</div>
   ...: </p>
   ...: </body>
   ...: </html>"""

In [2]: selector = scrapy.Selector(text=html)

In [3]: ''.join(selector.xpath('//div//text()').getall())
Out[3]: 'With the vision of providing creative sustainable solutions for global food crisis,\n    AquiNovo develops innovative, non-GMO, non-hormonal, peptide-based feed additives,\n    addressing the ever-growing demand for fish protein. Company’s additives improve both growth\n    performance and feed utilization, enabling the growth of more fish with less\n            feed. A unique peptide production system, enables large commercial\n    scale production at significant lower cost and carbon footprint. Growing more fish with less\n    feed also promote several SDG’s including the reduction of pressure on fish population in\n    the sea, providing food security and reducing hunger and poverty, climate change and\n    responsible production.\xa0'