• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Fixed rows while querying multiple .csv files|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Fixed rows while querying multiple .csv files|Power Query|Excel Forum|My Online Training Hub

vba course banner

Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryFixed rows while querying multiple …
sp_PrintTopic sp_TopicIcon
Fixed rows while querying multiple .csv files
Avatar
Gopalakrishnan M
Member
Members
Level 0
Forum Posts: 18
Member Since:
April 2, 2021
sp_UserOfflineSmall Offline
1
April 2, 2021 - 4:03 pm
sp_Permalink sp_Print

Thanks for the great post.

I am extracting data (through power query) from multiple .csv files. These individual .csv files (262 of them) are dynamically updated from one of the trading platform.
So one new row is appended for every 15 minutes in each of the .csv files (different stock symbols).

What I want is to maintain the same number of rows for each of these .csv files. In other words, as data (row) get appended to each of these .csv files, the number of rows keeps on increasing until end of the day and the following day the same routine. While I am aware of the Keep Bottom Rows, how can i do this for individual files (262 of them) through power query.

Here is the query :

let
Source = Folder.Files("C:\Yogya Data Collection\Data Between Dates"),
#"Split Column by Delimiter" = Table.SplitColumn(Source, "Name", Splitter.SplitTextByDelimiter(".", QuoteStyle.Csv), {"Name.1", "Name.2", "Name.3"}),
#"Removed Other Columns" = Table.SelectColumns(#"Split Column by Delimiter",{"Name.1", "Content"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Other Columns",{{"Name.1", "Symbol"}}),
#"Invoked Custom Function" = Table.AddColumn(#"Renamed Columns", "Transform File", each #"Transform File"([Content])),
#"Removed Columns" = Table.RemoveColumns(#"Invoked Custom Function",{"Content"}),
#"Expanded Transform File" = Table.ExpandTableColumn(#"Removed Columns", "Transform File", {"Date", "Time", "Open", "High", "Low", "Close", "Volume"}, {"Transform File.Date", "Transform File.Time", "Transform File.Open", "Transform File.High", "Transform File.Low", "Transform File.Close", "Transform File.Volume"}),
#"Renamed Columns1" = Table.RenameColumns(#"Expanded Transform File",{{"Transform File.Date", "Date"}, {"Transform File.Time", "Time"}, {"Transform File.Open", "Open"}, {"Transform File.High", "High"}, {"Transform File.Low", "Low"}, {"Transform File.Close", "Close"}, {"Transform File.Volume", "Volume"}})
in
#"Renamed Columns1"

Any help would be highly appreciated.

Much Regards,
~GK

sp_AnswersTopicSeeAnswer See Answer
Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4524
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
2
April 2, 2021 - 4:55 pm
sp_Permalink sp_Print

Hi GK,

Welcome to our forum!

If you want the action performed at the file level before appending the files together you make the transformations in the "Transform Sample File from..." query. All steps you apply in this query are applied to all files before appending.

Mynda

sp_AnswersTopicAnswer
Answers Post
Avatar
Gopalakrishnan M
Member
Members
Level 0
Forum Posts: 18
Member Since:
April 2, 2021
sp_UserOfflineSmall Offline
3
April 3, 2021 - 2:50 am
sp_Permalink sp_Print

Awesome Mynda, saved a lot of my time, been mulling over this since the past few days, really appreciate it.

Now you just opened up the can of worms, I am going bug you all so much with my queries now 🙂 (on a lighter side).

While I had paid a VB guy to get my data in a specific format, it is fine, but not too happy with the way and I love Power query.

If I want to extract certain time frame out of each file (each symbol, as each symbol is saved as a separate .csv file), to be more precise, let me say I want to get values (Open-high-low-close values) for each stock (or symbol), then how can i do this in power query.

I did try doing this, but the challenge is since the data is for two days (thanks to you, else I would be looking at zillions of rows), and when I chose any specific time frame for e.g. 15:15 and 15:30 hrs then I get the data (OHLC Values) for both the days for this time unit.  But I would want 15:15 and 15:30 hrs OHLC values only for yesterday and for today I will need data for let us say 9:15, 9:30 and 9:45 hours (OHLC values). How can I do this.

Let me know if you want me to attach the file and I shall do so.  (as I want to learn, I am not attaching the file).

Warm Regards
GK

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4524
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
4
April 3, 2021 - 2:42 pm
sp_Permalink sp_Print

Hi GK,

Glad I could help.

Please post your new question as a new topic and include a sample file. Note: please don't link the sample file to a CSV file as the source data. Just put the source data in the same Excel file as your query and be sure to show us your desired result so we know exactly what you're trying to achieve.

Mynda

Avatar
Gopalakrishnan M
Member
Members
Level 0
Forum Posts: 18
Member Since:
April 2, 2021
sp_UserOfflineSmall Offline
5
April 3, 2021 - 10:41 pm
sp_Permalink sp_Print

Thank you Mynda.  Will do so.

Cheers
GK

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Louis Muti
Guest(s) 8
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 873
Purfleet: 414
Frans Visser: 346
David_Ng: 306
lea cohen: 222
Jessica Stewart: 218
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Blair Gallagher
Brandi Taylor
Hafiz Ihsan Qadir
Gontran Bage
adolfo casanova
Annestine Johnpulle
Priscila Campbell
Jeff Mikles
Aaron Butler
Maurice Petterlin
Forum Stats:
Groups: 3
Forums: 24
Topics: 6369
Posts: 27852

 

Member Stats:
Guest Posters: 49
Members: 32359
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.