I have been picking up stock data from the web with power query for some time. Changed mid-May from Yahoo Finance (they changed data base structure) to Investing.com. Worked fine for a month, then last week, it will not update.
I have tried to create new query with https://se.investing.com/indices/us-spx-500-historical-data (as sample) but it does not connect to the address.
Anyone knowing why and if it can be solved? My settings or cookies? Or can Investing.com change so the data cannot be retrieved through queries?
Hi Joacim,
No idea why investing.com is not returning data, maybe they don't want to, some sites are using anti-bots to stop crawlers.
Looks like Google finance is showing the same data, why don't you use that: https://www.google.com/finance/historical?q=INDEXSP:.INX
Hi Catalin,
Investing.com have fx and futures, that is the reason I have been using them. I have started to use Google finance for the stocks.
So technically it’s possible for them to stop crawlers?
Yes, it's an old problem. Usually, a good firewall and site settings does most of the job, no need for specialized software like: https://www.shieldsquare.com/detect-and-block-bot-traffic/
Large sites do need such tools, but they should not block incoming connections.
You can try searching for their API documentation, you can use PQ to connect to their API Service, if they provide one. Most large sites do have an API service that can deliver clean data, no need to scrape the site.
Thank you Catalin!
Guess I need to open the "API" box again 🙂 , looked at it some time ago but as non-professional programmer it looked complicated ...
It's much easier in Power Query than before 🙂
