I am trying to scrape data from this website - Chartlink
I am able to do the basic scraping but not able to proceed with paginated scraping as url doesn't change upon changing the page.
I have found the api, and did proceed with Power Query too but could get the end result!
How did I get this : Right Click --> Inspect --> Network --> Fetch/XHR --> process
M Code:
let
url = "https://chartink.com/screener/process",
session = Web.Contents(url),
csrfToken = let Source = Web.BrowserContents(url) in
Html.Table(Source, {{"Column1", "meta[name=""csrf-token""]", each [Attributes][content]}})[Column1]{0},
condition = "{""scan_clause"": ""( {cash} ( latest rsi( 9 ) > latest wma( latest rsi( 9 ) , 21 ) and 1 day ago rsi( 9 ) <= 1 day ago wma( latest rsi( 9 ) , 21 ) and latest rsi( 9 ) > 70 and market cap > 1500 ) ) ""}",
headers = [#"Content-Type" = "application/json", #"x-csrf-token" = csrfToken],
source = Json.Document(Web.Contents(url, [Headers=headers, Content=Text.ToBinary(condition)]))
in
source
I get this error - DataSource.Error: Web.Contents failed to get contents from 'https://chartink.com/screener/process' (419)
Hi,
As the URL does not change as you go through the pages of data, that data is loaded using JavaScript and unfortunately can't be scraped using Power Query - or most other scraping methods without doing some complicated programming.
As for the API, I didn't see an API mentioned on their website, how did you get signed up to it?
I can't access the API link you provided, the website says it's a private link for the owner of the Scan only.
Without access to the API in the same way you have, I can't do anymore I'm afraid.
Regards
Phil
