Web Scraping From Linkedin

Posted on  by 



  1. Linkedin Screen Scraping
  2. Web Scraping From Linkedin Profile
  3. Scrape Linkedin Profile

Today I would like to do some web scraping of Linkedin job postings, I have twoways to go: - Source code extraction - Using the Linkedin API

I chose the first option, mainly because the API is poorly documented and Iwanted to experiment with BeautifulSoup.BeautifulSoup in few words is a library that parses HTML pages and makes it easyto extract the data.

Understanding the Document Object Model of Web Page For web scraping, we first have to understand what the real DOM of that web page is. So, go to the Yellow Pages and search for anything you want to. I will be searching for Software in Sydney. Press enter and you will see the result just like below. On September 9th, 2019 the UNITED STATES COURT OF APPEALS 1 has affirmed the former district court’s determination that a certain data analytic company is lawful to scrape perform automated gathering LinkedIn’s public profiles info. Scraping Solutions are industry specialists. With a fierce passion for data extraction and ethical web scraping services. We provide 100% managed services.

Official page: BeautifulSoup web page

Now that the functions are defined and libraries are imported, I’ll get jobpostings of linkedin.
The inspection of the source code of the page shows indications where to accesselements we are interested in.
I basically achieved that by ‘inspecting elements’ using the browser.
I will look for “Data scientist” postings. Note that I’ll keep the quotes in mysearch because otherwise I’ll get unrelevant postings containing the words“Data” and “Scientist”.
Below we are only interested to find div element with class ‘results-context’,which contains summary of the search, especially the number of items found.

Now let’s check the number of postings we got on one page

To be able to extract all postings, I need to iterate over the pages, thereforeI will proceed with examining the urls of the different pages to work out thelogic.

Linkedin Screen Scraping

Linkedin
  • url of the first page

  • https://www.linkedin.com/jobs/search?keywords=Data+Scientist&locationId=fr:0&start=0&count=25&trk=jobs_jserp_pagination_1

  • second page

  • https://www.linkedin.com/jobs/search?keywords=Data+Scientist&locationId=fr:0&start=25&count=25&trk=jobs_jserp_pagination_2

  • third page

  • https://www.linkedin.com/jobs/search?keywords=Data+Scientist&locationId=fr:0&start=50&count=25&trk=jobs_jserp_pagination_3

Web Scraping From Linkedin

Web Scraping From Linkedin Profile

there are two elements changing :
- start=25 which is a product of page number and 25
- trk=jobs_jserp_pagination_3

I also noticed that the pagination number doesn’t have to be changed to go tonext page, which means I can change only start value to get the next postings(may be Linkedin developers should do something about it …)

Scrape Linkedin Profile

As I mentioned above, all the information about where to find the job detailsare made easy thanks to source code viewing via any browser

Next, it’s time to create the data frame

Now the table is filled with the above columns.
Just to verify, I can check the size of the table to make sure I got all thepostings

In the end, I got an actual dataset just by scraping web pages. Gathering datanever have been as easy.I can even go further by parsing the description of each posting page andextract information like:
- Level
- Description
- Technologies

There are no limits to which extent we can exploit the information in HTML pagesthanks to BeautifulSoup, you just have to read the documentation which is verygood by the way, and get to practice on real pages.

Ciao!





Coments are closed