Hi all! As my excel project continues to grow, my power query tables are getting monstrous.
What I typically do is append tables, add some columns, do some math, then export it into a new sheet as hard values to increase speed, rinse, and repeat. This year on my projects, I've been focusing on more vba and such which has allowed me to gather my data quickly via web query and power query but has also made me greedy for more data which is getting cumbersome for my pc to handle. I'm not sure if it's my pc or excel in general.
What I'm asking is at what point is it time to move on and store my data now into a database and use excel to interact with that data base (assuming that is possible?) I use compact sql for my work stuff, but it's set up in such away that I have to do little under the hood manipulation with it as that is done via my work software. I've read about access a bit and have it as part of 365 but am not sure if that will help speed up performance or not. Mynda have any helpful videos on using access with excel? I'm going to have a look here in a bit.
Right now my current work book is 45000KB. I'm working with 2 imported appended power query tables
one is 11 wide by 465000 tall and the other is 166 wide by 251000 tall. On my simulation Workbook when I do a full sim it does about 61 billion random numbers and some sum cells in 20 seconds, but now were talking almost 117billion cells of some math lovin. The big thing that is bogging it down is an index match (which I understand to the best of my knowledge) which is supposed to be faster to calc. than vlookup? Besides, I need the additional criteria to form my "bridge" between the two tables.
I guess what has me a bit puzzled is why the size of a work sheet is super duper big if excel isn't meant to handle it?
Any advice / feed back is appreciated.
Thanks
Matthew
In my mind, if you have a data base then you should be using Data Base Software (ie. MS Access) as spreadsheets are really not the most effective way to maintain it. Spreadsheets are great for calculating and analysis but are cumbersome for data base. My 2 cents.
BTW: Here is a link to start if you have never used Access--> http://www.accessmvp.com/strive4peace/
Right on Alan thanks so much.
For practice I started creating a database for my nfl football model
was able to create a schedule table
team tables with composite keys
and one years worth of rosters and managed to create relationships between the 3.
I like how smooth it flows once you get comfortable with the navigation along with how snappy the drop down lists fire off .
Going to play around with this some more and see what all I can do within it and how I can use it more with excel and vice versa before I integrate other projects and work related material
thanks!
https://screencast-o-matic.com/watch/crXYj3VIuDe
Matthew
Here is a white paper that will help you as you go forward.
