• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Get ALL Entries from a Webpage|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Get ALL Entries from a Webpage|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryGet ALL Entries from a Webpage
sp_PrintTopic sp_TopicIcon
Get ALL Entries from a Webpage
Avatar
Matthias
Member
Members
Level 0
Forum Posts: 48
Member Since:
December 13, 2017
sp_UserOfflineSmall Offline
1
August 25, 2022 - 2:25 am
sp_Permalink sp_Print

Hello,

I would like to scrape a webpage with power query. When I open the page in a browser it shows me 50 entries and only if a scroll to the bottom additional entries are shown. It is not additional pages, but showing the addional entries on the same page with the same url.

In power query I also see only 50 entries, but here I can't scroll down to get additional entries.

Adding time like this (e.g. 50sec), does not help to get all entries - still only 50 entries are shown, resp. included in the html code:
Web.BrowserContents("https:/...", [WaitFor = [Timeout = #duration(0,0,0,50)]])

Any chance to force a webpage to deliver all entries?

Regards,
Matthias

Avatar
Riny van Eekelen
Örnsköldsvik, Sweden
Moderator
Members


Trusted Members

Moderators

Power BI
Level 0
Forum Posts: 441
Member Since:
January 31, 2022
sp_UserOfflineSmall Offline
2
August 25, 2022 - 3:08 am
sp_Permalink sp_Print

Hi Matthias,

I would be helpful if you could share the link the web-site, provided it doesn't contain confidential information.

Riny

Avatar
Matthias
Member
Members
Level 0
Forum Posts: 48
Member Since:
December 13, 2017
sp_UserOfflineSmall Offline
3
August 25, 2022 - 4:18 am
sp_Permalink sp_Print

Hi Riny,
it is a webpage, which contains info, which I should not share.
And now I only find webpages which work with page numbering e.g. Blog • Page 2 of 61 • My Online Training Hub
But it is not rare that webpages initially only give a subset and after scrolling down add entries, so it should be possible to find some example. Perhaps you know one / can find one too.

Thanks,
Matthias

Avatar
Riny van Eekelen
Örnsköldsvik, Sweden
Moderator
Members


Trusted Members

Moderators

Power BI
Level 0
Forum Posts: 441
Member Since:
January 31, 2022
sp_UserOfflineSmall Offline
4
August 25, 2022 - 5:01 am
sp_Permalink sp_Print

Perhaps you'll find an solution here:

https://www.myonlinetraininghu.....ower-query

Avatar
Philip Treacy
Admin
Level 10
Forum Posts: 1515
Member Since:
October 5, 2010
sp_UserOfflineSmall Offline
5
August 25, 2022 - 10:16 am
sp_Permalink sp_Print

Hi Matthias,

PQ can't interact with a website in this way.  You could use Power Automate though.

You can start a browser, then send the END key to the browser window to make it scroll to the end of the page and load more data.

You then of course need to create the PA steps to scrape the data you need.  You can't use PA and PQ together to scrape data.

Regards

Phil

Avatar
Matthias
Member
Members
Level 0
Forum Posts: 48
Member Since:
December 13, 2017
sp_UserOfflineSmall Offline
6
August 26, 2022 - 3:18 am
sp_Permalink sp_Print

Thanks Riny, that can only be used if the url is changing for each page, or every 10 entries.

Phil, "You can't use PA and PQ together to scrape data" means there is no way with PQ. I hoped for some kind of url modification like ?sort=desc or a kind of filter added to the url which could make the query work incremental.

Thanks,
Matthias

Avatar
Philip Treacy
Admin
Level 10
Forum Posts: 1515
Member Since:
October 5, 2010
sp_UserOfflineSmall Offline
7
August 26, 2022 - 8:05 am
sp_Permalink sp_Print

Hi Matthias,

The behaviour you describe from the webpage indicates that the loading of new data is triggered by scrolling to the bottom of the page. as such it must be triggered by JavaScript.  Unfortunately there is no way for PQ to make web pages scroll like this and/or trigger JS.  But Power Automate can.

If you could load more data by adding a parameter to the URL then you could use PQ, but it doesn't sound like this is possible from what you have said, and without being able to access the webpage, I can't check.

Regards

Phil

Avatar
Matthias
Member
Members
Level 0
Forum Posts: 48
Member Since:
December 13, 2017
sp_UserOfflineSmall Offline
8
August 26, 2022 - 11:53 pm
sp_Permalink sp_Print

Hi Phil,

yes,  it is a typical scroll to the bottom to load more data behaviour and the code mentions type="text/javascript".
I can use ?order=XYZ to sort the data (ascending) according to different criteria. There might be other parameters, but that is all what I found.

Thanks again,
Matthias

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Ngoc Tinh
Guest(s) 9
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 870
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
A.Maurizio: 202
Jessica Stewart: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
michael serna
mashal sana
Tiffany Kang
Leah Gillmore
Sopi Yuniarti
LAFONSO HERNANDEZ
Hayden Hao
Angela chen
Sean Moore
John Chisholm
Forum Stats:
Groups: 3
Forums: 24
Topics: 6215
Posts: 27248

 

Member Stats:
Guest Posters: 49
Members: 31897
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.