• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Extremely slow web query|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Extremely slow web query|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryExtremely slow web query
sp_PrintTopic sp_TopicIcon
Extremely slow web query
Avatar
Ron Jawes

Active Member
Members
Level 0
Forum Posts: 5
Member Since:
April 1, 2022
sp_UserOfflineSmall Offline
1
April 2, 2022 - 6:48 am
sp_Permalink sp_Print

Hello,

My elderly father loves numbers and greyhound racing. Watching him use pen and paper to copy information from a website was painful. I thought I would have a go at automating it for him. I have some experience with excel but I am brand new to power query. After many attempts I was able to produce results but they are extremely slow, to the point where pen and paper was faster.

I can produce a table of a single dog's racing history in around 20 seconds. As soon as I created a list of all dogs running for the day, the time increased to several minutes per dog. This would take more than 24hrs to run each day.

  1. Create a table of URLs of the meetings for the day. I failed at being able to accomplish this with power query, however I was able to do it by downloading Power BI and using "Add table using examples" (getTodayFieldsURLsMcode.txt)
  2. Export table to a csv file (data.csv)
  3. Create a table of runners/greyhounds that are racing today (TodayFullFieldListMcode.txt) 
  4. Create a table of all runners/greyhounds racing history (TestTableMcode.txt)
  5. I invoked two functions to loop through the tables above (fxTodayRunnerNamesMcode.txt, fxCombinedMcode.txt)

If anyone with greater experience and knowledge than me could guide me through my errors, I would greatly appreciate it.

sp_AnswersTopicSeeAnswer See Answer
Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1810
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
2
April 4, 2022 - 4:53 pm
sp_Permalink sp_Print

Hi Ron,

can you upload your PBI model?

Avatar
Ron Jawes

Active Member
Members
Level 0
Forum Posts: 5
Member Since:
April 1, 2022
sp_UserOfflineSmall Offline
3
April 4, 2022 - 9:02 pm
sp_Permalink sp_Print

Hello Catalin,

Thank you for your reply.

Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1810
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
4
April 8, 2022 - 4:12 pm
sp_Permalink sp_Print

Hi Ron,

Checked your model, but looks like it does not have the codes you uploaded in the first post. It's hard to rebuild your entire logic, can you provide a functional model that does produce results, even if it's slow?

Avatar
Ron Jawes

Active Member
Members
Level 0
Forum Posts: 5
Member Since:
April 1, 2022
sp_UserOfflineSmall Offline
5
April 9, 2022 - 11:16 am
sp_Permalink sp_Print

Hi Catalin,

Thanks again for your time. I did it all through Excel/Power Query. The only part I couldn't do there I was able to achieve with Power BI.

The text files above are the Mcode I used in Excel to create the final table. I am unsure what else I can upload. Should I post a link to the website along with a sample table of what I want?

Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1810
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
6
April 9, 2022 - 2:00 pm
sp_Permalink sp_Print

Please upload the excel file with your queries.

Thank you

Avatar
Ron Jawes

Active Member
Members
Level 0
Forum Posts: 5
Member Since:
April 1, 2022
sp_UserOfflineSmall Offline
7
April 9, 2022 - 5:04 pm
sp_Permalink sp_Print

Thank you for your patience Catalin.

Files attached as requested

Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1810
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
8
April 10, 2022 - 3:51 pm
sp_Permalink sp_Print

Hi Ron,

When you do a query from web, Power Query is using Web.Page function that converts the text into html, the source step looks like this:

= Web.Page(Web.Contents("https://www.thegreyhoundrecorder.com.au/form-guides/"))

If you remove the Web.Page function, you will see a binary object in PQ Editor. Right click that binary and choose Text, not HTML.

You'll be able to see the content of the page, in plain text, just apply filters in that column to get what you need, see the below query :

let
Source = Table.FromColumns({Lines.FromBinary(Web.Contents("https://www.thegreyhoundrecorder.com.au/form-guides/"), null, null, 65001)}),
#"Filtered Rows1" = Table.SelectRows(Source, each (Text.Contains([Column1] ,"/form-guides/") and Text.Contains([Column1] ,"/fields/"))),
#"Extracted Text After Delimiter" = Table.TransformColumns(#"Filtered Rows1", {{"Column1", each Text.AfterDelimiter(_, "<td><a href="""), type text}}),
#"Extracted Text Before Delimiter" = Table.TransformColumns(#"Extracted Text After Delimiter", {{"Column1", each Text.BeforeDelimiter(_, """><button"), type text}})
in
#"Extracted Text Before Delimiter"

Unfortunately, can't see a way to speed up the queries.

sp_AnswersTopicAnswer
Answers Post
Avatar
Ron Jawes

Active Member
Members
Level 0
Forum Posts: 5
Member Since:
April 1, 2022
sp_UserOfflineSmall Offline
9
April 10, 2022 - 7:01 pm
sp_Permalink sp_Print

Dear Catalin,

Thank you very much for sharing your time and knowledge. You have taught me more than I knew before posting. I hope others also learn from your teachings. If I ever get this fully working, I will post an update.

Thank you to all Admin for a wonderful platform and community.

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Anders Sehlstedt, RAQUEL ACION, Dana Friedt, Chandler Davis, gogy13, Brenda Richards, Mark Luke
Guest(s) 10
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 870
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
Jessica Stewart: 202
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Mark Luke
terimeri dooriyan
Jack Aston
AndyC
Denise Lloyd
michael serna
mashal sana
Tiffany Kang
Leah Gillmore
Sopi Yuniarti
Forum Stats:
Groups: 3
Forums: 24
Topics: 6218
Posts: 27267

 

Member Stats:
Guest Posters: 49
Members: 31901
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.