• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Combining CSV files; using file name for information that is not present in the actual CSV file|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Combining CSV files; using file name for information that is not present in the actual CSV file|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryCombining CSV files; using file nam…
sp_PrintTopic sp_TopicIcon
Combining CSV files; using file name for information that is not present in the actual CSV file
Avatar
R G

Active Member
Members
Level 0
Forum Posts: 3
Member Since:
February 14, 2020
sp_UserOfflineSmall Offline
1
February 14, 2020 - 3:03 am
sp_Permalink sp_Print

I'm trying to combine the following three CSV files in power query.

These three files have the same columns. The CSV files do not contain the SN# and POLE information shown below - I got these from their file names.

When I click on each individual table to see its attributes, all three show their respective data, but after I expand the table attributes, only the information from the first file shows up. The other two files all have null values.

sfsdfd

Annotation-2020-02-13-115316.pngImage Enlarger

Here is the mcode:

 

let
    Source = #"INCON DATA",
    #"Filtered Rows" = Table.SelectRows(Source, each ([AREA] = "sf6-daily-summary.csv")),
    #"Filtered Hidden Files1" = Table.SelectRows(#"Filtered Rows", each [Attributes]?[Hidden]? <> true),
    #"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File from INCON DATA SF6-DAILY", each #"Transform File from INCON DATA SF6-DAILY"([Content])),
    #"Removed Other Columns1" = Table.SelectColumns(#"Invoke Custom Function1",{"SN#", "Pole", "AREA", "Date created", "Transform File from INCON DATA SF6-DAILY"}),
    #"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File from INCON DATA SF6-DAILY", Table.ColumnNames(#"Transform File from INCON DATA SF6-DAILY"(#"Sample File"))),
    #"Changed Type" = Table.TransformColumnTypes(#"Expanded Table Column1",{{"DATE_STAMP", type date}, {"AVG_DENSITY_A", type number}, {"AVG_PRESSURE_A", type number}, {"AVG_TEMP_A", type number}, {"DENS_TREND_A", type number}, {"PRESS_TREND_A", type number}, {"TREND_CONF_A", Int64.Type}, {"AVG_MASS_A", type number}, {"MASS_CHANGE_A", Int64.Type}, {"ACCUM_MASS_LOSS_A", Int64.Type}})
in
    #"Changed Type"

 

Can anybody help me out?

 

Thanks,

R

sp_PlupAttachments Attachments
  • sp_PlupImage Annotation-2020-02-13-115316.png (306 KB)
sp_AnswersTopicSeeAnswer See Answer
Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4449
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
2
February 14, 2020 - 11:48 am
sp_Permalink sp_Print

Hi and welcome to our forum!

It looks like the very first thing you do (second line of code) is filter the source to only return the file called sf6-daily-summary.csv which will be why you're only getting the data from one file. i.e. you've only chosen one file. If you remove this step do you get all of the data?

Mynda

Avatar
R G

Active Member
Members
Level 0
Forum Posts: 3
Member Since:
February 14, 2020
sp_UserOfflineSmall Offline
3
February 14, 2020 - 2:49 pm
sp_Permalink sp_Print

Hi Mynda:

First, thank you for looking into this.

Unfortunately, removing that step did not get all of the data.

Also, this step is filtering several other "types" of CSV files that have dissimilar column headers. Please see the image below for clarification.

Annotation-2020-02-13-232615.pngImage Enlarger

Moreover, the fifth step is where I'm running into trouble, I think. Up to this step, data for all three tables shows when I click on the separate tables (see below for example). It is when I try to expand the tables that data from the first file's table is the only data that shows up.

Annotation-2020-02-13-233314-1.pngImage Enlarger

Annotation-2020-02-13-233347.pngImage Enlarger

Annotation-2020-02-13-233404.pngImage Enlarger

After I click on expand table...[added date filter for simplification] only data from the first file shows up.

#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File from INCON DATA SF6-DAILY", Table.ColumnNames(#"Transform File from INCON DATA SF6-DAILY"(#"Sample File"))),

Annotation-2020-02-13-234142.pngImage Enlarger

Again, I appreciate the help with this.

 

Thanks,

R

sp_PlupAttachments Attachments
  • sp_PlupImage Annotation-2020-02-13-232615.png (54 KB)
  • sp_PlupImage Annotation-2020-02-13-233314.png (250 KB)
  • sp_PlupImage Annotation-2020-02-13-233404.png (266 KB)
  • sp_PlupImage Annotation-2020-02-13-233314-1.png (250 KB)
  • sp_PlupImage Annotation-2020-02-13-234142.png (20 KB)
Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4449
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
4
February 14, 2020 - 3:05 pm
sp_Permalink sp_Print

Hi, the images don't help a lot because in one of them I don't know what step they're showing. What is the source of the data? It appears to be another query. I don't see how you can have multiple files in a folder with the same file name, which is what appears to be the case in one of the screenshots.

Perhaps you can share your files, the query file and the 3 CSV files you want to import. This will be far more helpful than screenshots.

Mynda

Avatar
R G

Active Member
Members
Level 0
Forum Posts: 3
Member Since:
February 14, 2020
sp_UserOfflineSmall Offline
5
February 15, 2020 - 1:48 am
sp_Permalink sp_Print

Mynda,

I've attached the three CSV files and the excel file which has the query.

This query only has the three files loaded, but it doesn't seem to get the data from the second and third file.  It shows an error this time.

Thanks,

R

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4449
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
6
February 15, 2020 - 9:40 am
sp_Permalink sp_Print

Hi,

Thanks for sharing the files. The 3 CSV files have different column headers. This is the problem. P1 column headers end in _A, P2 end in _B and P3 end in _C

They must all have identical column names. I assumed you had already checked this because you said initially that "these three files have the same columns."

Mynda

sp_AnswersTopicAnswer
Answers Post
sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: AndyC
Guest(s) 9
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 871
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
A.Maurizio: 202
Jessica Stewart: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Raj Mattoo
Mark Luke
terimeri dooriyan
Jack Aston
AndyC
Denise Lloyd
michael serna
mashal sana
Tiffany Kang
Leah Gillmore
Forum Stats:
Groups: 3
Forums: 24
Topics: 6219
Posts: 27276

 

Member Stats:
Guest Posters: 49
Members: 31903
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.