Job Crawler

Our Experience developing an automated data crawler capable of bypassing captcha.

Customer Profile:

Our client is one of the leading data providers based out of Japan. They specialize in indexing data from job portals and identify the company’s tech expertise in real-time. Our client is a pioneer in analyzing and structuring company buying intent based on historical tech hirings. With millions of real-time updates, their customers uncover stealth tech start-ups or hiring needs of their network to engage in real-time with the right context. This helps them to get structured demand in tech, overall company trend, and help to predict the next move.

Problem Statement:

Getting a structured database and real-time updates of the job posts in scale while staying ahead of the competition can be a complex task. An automated crawler that can provide valid and structured data based on the current market needs is a need of the hour.

Project Background:

Our client is a data provider in need of contact information for the job posts in different external platforms. They needed a data crawler that helps to mine all the required data of the job posts and want to make sure the data provided is always up to date.

Challenges:

  • Automatically bypassing the captcha.
  • Non-Uniform Structures.
  • Maintain Database Freshness.
  • Bandwidth and Impact on Web Servers.
  • Absence of Context.

Technology:

Development language: Java Database Server: MySQL File Storage: AWS S3 Hibernate Content Delivery: AWS CloudFront

Solution:

This client’s main concern was turnaround time on this project and getting real-time details of all the job postings. We built an expert team of professionals for this project to deliver in time with the highest accuracy level. They curated a crawler accessible through the web and capable of scrapping the job post depending on the client’s needs. The process is executed in two steps; Initially, the details from the listing page are scraped based on pre-set parameters. Then the scrapped list data is again imported into the system to get detailed information on each job post using the job post ID as the primary identifier. For maximum accuracy, we implemented an automated QA check to verify the records generated by the crawler to avoid duplication and redundant data. These efforts were appreciated in a great manner by the client.

Achievements:

  • Achieve Automation
  • Business Intelligence & Insights
  • Unique and rich datasets
  • Effective Data Management

Return on Investment:

The fully automated data crawler increased the operational efficiency by 35%