Log In
Register
Home
New Member
July 18, 2024
Offline
I am trying to scrape data from this website - Chartlink
I am able to do the basic scraping but not able to proceed with paginated scraping as url doesn't change upon changing the page.
I have found the api, and did proceed with Power Query too but could get the end result!
How did I get this : Right Click --> Inspect --> Network --> Fetch/XHR --> process
M Code:
let
url = "https://chartink.com/screener/process",
session = Web.Contents(url),
csrfToken = let Source = Web.BrowserContents(url) in
Html.Table(Source, {{"Column1", "meta[name=""csrf-token""]", each [Attributes][content]}})[Column1]{0},
condition = "{""scan_clause"": ""( {cash} ( latest rsi( 9 ) > latest wma( latest rsi( 9 ) , 21 ) and 1 day ago rsi( 9 ) <= 1 day ago wma( latest rsi( 9 ) , 21 ) and latest rsi( 9 ) > 70 and market cap > 1500 ) ) ""}",
headers = [#"Content-Type" = "application/json", #"x-csrf-token" = csrfToken],
source = Json.Document(Web.Contents(url, [Headers=headers, Content=Text.ToBinary(condition)]))
in
source
I get this error - DataSource.Error: Web.Contents failed to get contents from 'https://chartink.com/screener/process' (419)
October 5, 2010
Offline
Hi,
As the URL does not change as you go through the pages of data, that data is loaded using JavaScript and unfortunately can't be scraped using Power Query - or most other scraping methods without doing some complicated programming.
As for the API, I didn't see an API mentioned on their website, how did you get signed up to it?
I can't access the API link you provided, the website says it's a private link for the owner of the Scan only.
Without access to the API in the same way you have, I can't do anymore I'm afraid.
Regards
Phil
1 Guest(s)
