Example: Links by Condition

In this example, we will demonstrate how to use the InstantAPI.ai Retrieve API endpoint to scrape links from a web page that meet a specific condition you set. This in particularly helpful when recursively scraping web pages throughout a website, but you wish to limit the crawl to the kinds of web pages you require. We will provide a complete request and explain what each part of the request does. Additionally, we will show the expected response structure.

Complete Request

Here is a complete request to the Retrieve API endpoint:

Request Breakdown

  • webpage_url: The URL of the web page you want to scrape.
  • api_method_name: A user-defined name for the API action, in this case, “getIndividualProductURLs”.
  • api_response_structure: The expected structure of the APIs response, defined by you. This includes placeholders for the details you want to scrape.
  • link_extract: Turns the link extract mode on.
  • api_key: Get your API key.

Expected Response Structure

The expected response structure is defined in the api_response_structure parameter. Here is the formatted version for clarity:

{
  "urls": [
    {
      "url": ""
    }
  ],
  "next_page_url": ""
}

Example Response

When the request is successfully processed, you can expect a response similar to the following:

{
  "urls": [
    {
      "url": "https://www.templeandwebster.com.au/Pure-French-Flax-Linen-Fitted-Sheet-JHLF-TMPL2720.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Luxury-1000TC-Cotton-Blend-Sheet-Set-TMPL1635.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/400TC-Bamboo-and-Cotton-Sheet-Set-SUBSSW-TMPL3570.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Lexi-Cotton-Fitted-Sheet-and-Pillowcase-Set-TMPL5488.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Vintage-Washed-Sheet-Set-TMPL5294.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Gingham-Washed-Cotton-Sheet-Set-GIOI1163.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Kirby-Cotton-Flannelette-Sheet-Set-TMPL5823.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Laila-Vintage-Washed-Sheet-Set-TMPL6118.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Heston-Cotton-Fitted-Sheet-and-Pillowcase-Set-BICN2191.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/500TC-Bamboo-and-Cotton-Fitted-Sheet-Set-PAVE1096.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Organic-Washed-Cotton-Sheet-Set-LALN1002.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Washed-Microfibre-Sheet-Set-DRMK1828.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Washed-Microfibre-Fitted-Sheet-Set-DRMK1844.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/400TC-Bamboo-Sheet-Set-NATB1046.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Brett-Vintage-Washed-Microfibre-Sheet-Set-CPLY1032.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Park-Avenue-Bamboo-and-Cotton-Sheet-Set-PAVE1069.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Vienna-Cotton-Fitted-Sheet-with-40cm-Wall-0172-LNHO2697.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Heston-300TC-Cotton-Percale-Sheet-Set-BICN2089.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Bamboo-and-Microfibre-Sheet-Set-ROLC1179.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Harriet-Bamboo-and-Cotton-Fitted-Sheet-Set-GIOI1029.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Soft-Touch-Waterproof-Cot-Fitted-Sheet-DRMK1612.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/European-Flax-Linen-Sheet-Set-70072-NATB1038.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/1000TC-Egyptian-Cotton-Sheet-Set-LALN1032.html"
    },
    {
      "url": "https://www.templeandwebster.com.au/Park-Avenue-Egyptian-Cotton-Flannelette-Sheet-Set-PAVE1068.html"
    }
  ],
  "next_page_url": "https://www.templeandwebster.com.au/Bed-Sheets-C1813085.html?itemsperpage=24&sortby=6&curpage=2"
}

By following this example, you can easily use InstantAPI.ai to scrape and transform data from various web pages into structured data tailored to your needs.