When bootstrapping <a href="https://www.nestpick.com/" target="_blank">Nestpick</a> in 2016, I decided to work on a web scraping and search playground so I could learn and prototype the features that we would be adding to the product later. Nestpick is an apartment mid-term rental aggregator that allow users to search on several providers at once. The technical solution relies on scraping these partners via API or web and indexing this data on a search engine component.
I partnered again with <a href="https://twitter.com/lsdr" target="_blank">Luiz Rocha</a>, a long time friend, to build this playground using <a href="https://scrapy.org/" target="_blank">Scrapy</a> and <a href="https://solr.apache.org/" target="_blank">Solr</a>.
<blockquote>
<a href="https://github.com/spare-time/pinboogle#pinboogle" target="_blank">The code is available at Github</a>
</blockquote>
The project is structured in X parts:
<ul>
<li>Web scraper that fetches the content of all bookmarked links in <a href="https://pinboard.in/" target="_blank">Pinboard.in</a>, using Scrapy.</li>
<li>Index that is stored in Solr search engine</li>
<li>A search frontend built in python Flask framework that work as a Solr client.</li>
</ul>
The project README shows you how to setup it yourself so feel free to use it in case you need to play around search engines and web scraping.

A search playground with Pinboard web scraping and Solr search engine deployed on docker containers.