Let’s say that part of your lead generation approach is searching and collecting contact data from for sale by owner websites.
If you do this day by day, it can become a drag and consume a lot of time. And if you did some research on the web about how you could automate this task, you might have come across the term “web scraping”.
Then, you might wonder what is scraping in real estate?
It is the automated process that researches one or more particular websites for you, extracts data, and re-organizes it in a fashion that you can use for another purpose.
Especially in real estate marketing, web scraping can help you a lot when it comes to lead generation by extracting contact information from particular pages.
In this article, I will discuss more in-depth what web scraping is, examples of web scraping in real estate, challenges that web scrapers have to deal with, the general benefits of web scraping, and four providers that offer web scraping solutions.
What Is Web Scraping?
In a sense, you can consider web scraping an automated process that does the research part for certain data on particular websites for you.
It is a kind of digital assistant that extracts specific data from websites for you and then displays it in an organized manner.
The process usually goes like this:
1) You go to the respective web scraper and define what websites need to be visited and what type of data needs to be extracted.
2) The program or script executes and visits the websites and extracts the relevant data.
3) The extracted data is stored.
4) The stored data is displayed in an organized manner. This can be on a results page of the respective scraper program.
In this context, there are also additional options possible, such as sending you the organized data to your inbox or adding it to a third party software such as a customer relationship management software in the case of potential real estate leads.
So, in general, a web scraper extracts certain data from a website in order to make use of it for another purpose somewhere else.
Examples of Web Scraping in Real Estate
There are many different use cases possible for web scraping in Real Estate and, most interesting to me, there are also a bunch in the context of real estate marketing.
You can use web scraping to extract data that will give you additional information about real estate market trends, pricing trends, or competitors.
Another use case is the monitoring of vacancy rates, estimating rental yields, and appraising property value.
You could have a web scraper to scrape real estate listing data from a partner agency to list certain properties on your website.
This might be necessary when an internal solution, such as, for example, an IDX integration is not a viable option.
The last one that comes to mind and that is of my highest interest of course, is the use of web scraping for real estate lead generation.
For real estate pros, the following web scraper use cases in the context of lead generation could be helpful:
- Scraping contact data from sellers of for sale properties by owner websites.
- Scraping contact data from potential investors of rental property listing websites.
- Scraping contact data from sellers on listing portals with expired listings.
- Scraping the contact data from potential realtor partners for co-brokerage opportunities in other regions.
- Scraping contact data from potential sellers of commercial real estate by using Google Maps.
Scraping contact data from certain courthouses to extract contact data from potentially motivated sellers.
In terms of then displaying and presenting the extracted data for lead generation, basically, the sky’s the limit.
The algorithm used could, for example, also include your monthly revenue goals and the output of the number of leads you would need to contact per day to reach these goals based on current conversion rates depending on the medium you use to contact them.
Challenges of WebScrapers in Real Estate (But Not Limited to It)
Some websites, such as Craigslist use methods that prevent web scraping. They, for example, disallow bots from crawling.
In these cases, the respective web scraper needs to be a bit more sophisticated and be able to simulate human browsing.
Another challenge can be if the webpage to be scraped has a changeable and complicated web page structure.
So, you can’t build just one scraper for many different websites. You will usually need to build one scraper for each website.
Another issue is IP blocking. The IP of the web scraper is blocked when the to-be-scraped website receives too many requests in a certain amount of time.
For that reason, the scraper might need to use an IP proxy solution.
The next one is of course the famous CAPTCHA. You know the question you sometimes see when using a form that asks you if you are human.
Most web scrapers can’t circumvent this just yet.
Some scrapers stop then and some manual CAPTCHA input from the user (you) is necessary before they can continue.
Then, there are also so-called honeypot traps. Sounds sweet, but it’s not that sweet for less sophisticated scrapers.
A honeypot trap is a trap that is implemented on the to-be-scraped website to catch a scraper.
It usually involves some links invisible to humans but visible to scrapers.
When the scraper catches them, the website starts to block the scraper.
There are a few more challenges scraper developers need to take into consideration for a well-working scraper.
But you can see that it’s not necessarily an easy endeavor to have a well-working script in this context.
The Benefits of Web Scraping
Web scraping in general and thus also for real estate has many advantages.
I consider the two major benefits of web scraping for real estate saving time and saving costs.
Let’s use the case of finding seller contact information on different property listing websites.
Without a scraper, you or your assistant would have to spend hours on different property listing websites and copy and paste contact information, maybe in an Excel sheet.
With a great web scraper, you won’t need to invest this time or your money (in the case of an assistant) to get this contact information.
Provided it is working well, you could wake up each morning to a new batch of potential leads in your inbox or in your customer relationship management software that was added automatically by the respective scraper.
When it comes to strategic decision-making in real estate marketing, a scraper could also be on the lookout for current marketing key performance indicators of different online marketing channels and generate statistics.
Based on these statistics, you could make the decision on which marketing channel to focus next.
4 Web Scraping Tools You Could Also Use for Real Estate
During my research, I found out that most scrapers on the market have a rather generic approach, but can be tailored to real estate use cases.
What does this mean?
Although you won’t have to develop your own real estate web scraper, it means that you will still need some tech affinity to adapt and tailor the different software solutions to your individual needs as a realtor or as another real estate pro.
Below I collected four scrapers for you that can also be used for or tailored to real estate needs.
ParseHub presents itself as a scraper tool for every need. Their main features include the following:
- Scraping new sales leads from directories, communities and social media
- An API to build mobile and web apps
- Aggregation of data from different websites into one platform
- Extracting products and prices from online retailers (e-commerce)
- Scheduled runs
- Automatic IP rotation
- Text, HTML, and attributes can be extracted
- Images/files download
- Data extraction behind a login page
- Downloadable CSV and JSON files
- Dropbox integration
- Data extractable from tables and maps
- Browser-based, graphic interface
ParseHub has four different pricing plans.
The free plan allows you to scrape 200 pages, but doesn’t have any IP rotation or scheduling functionality.
The ones that do have this functionality are the paid plans that are between $149 and $499 per month.
You can learn more about the company here.
DataOX is rather a data service powered by scraping software, rather than a software that you could use on your desktop or as a web application.
They call themselves a web scraping and data aggregation service, and provide this service in the following areas:
- Flight monitoring
- Job posts
- Brand monitoring
- Real estate
- Lead generation
- Machine learning
- Price monitoring
- Review scraping
- Document scraping
- URL scraping
- Data scraping for e-commerce
- Amazon scraping
- Ali Express scraping
- Social media
- Public sources
- Google trends
- Financial data
- Sports data
Their pricing plans are both on a per delivery basis and on a monthly basis.
For a one-time data delivery, you pay $300 and if you need scraping data on a monthly basis, prices start at $250.
You can learn more about DataOx here.
DataHut is another data extraction or web scraping service provider. Their focus is on customized solutions according to your needs.
So, it’s not a scraping software that is focused on certain areas and websites to extract data from, but according to what you need to scrape, they tailor a scraping solution for you.
DataHut has a three-tier pricing plan.
It starts with the personal plan for $40 per month per website, and you get up to 10,000 records per month and a weekly or monthly crawling frequency.
The next one is the business plan for $100 per website per month. With this plan, you get 100,000 records per month and a daily, weekly or monthly crawling frequency.
For 100% tailored solutions, no pricing information was available, since this all depends on your individual needs and the scope of work that would need to be done.
You can learn more about DataHut here.
WebScrapingApi is again more a customizable scraping software than a scraping service.
This means more involvement on your behalf would be necessary to make the right configuration of the tool so it does what you want.
But it would also require slightly more tech affinity.
WebScrapingApi’s scraping software is developed in a way that it can manage all sorts of possible blocking points (e.g. proxies or CAPTCHAs).
You can customize requests by modifying IP geolocation, headers, and more.
When it comes to pricing, the software company offers a free plan with 5,000 API calls. The “Grow” plan costs $90 per month and includes 1,000,000 API calls.
You can learn more about WebScrapingApi here.
Since I have a bit of a coding and web development background, I could understand what most of the scraping solution providers have to offer.
But I doubt that the average real estate professional with less or no technical background would understand most of the rather general solutions on the market.
By being rather generic, these solutions often require more assistance and tailoring on behalf of the provider which, in turn, would likely mean higher costs for you.
Additionally, these are often solutions where web developers can program on top of their already existing coding base and application programming interface (API).
I have no doubt that web scraping can give you a nice edge in your real estate business when it comes to lead generation and other processes where usually repetitive tasks and a certain time investment is involved to gather and organize data from the web.
However, I wish I had found a more niched down solution that is more tailored for realtors and other real estate professionals.