Pies, Lies and AIs
Showing the struggle of organisations to address the data revolution

Pies, Lies and AIs
Exploring the world of data and organisational intransigence
Geolocation in Fabric Real Time Intelligence
Lots of announcements at Build 2024. It's easy to miss the significant ones in lieu of the noise of the multitude of stuff. I've been...
The new new normal
Well the number of people looking for work in the cloud and data sector has now shot up as everyone adjusts their forecast expectations...
Ready, steady, gone: The epic battle with keeping your devs happy
The nature of Elastacloud is beginning to change. Most of my advisors and sales team have never written a line of code in their life and...
Councils need to think about Carbon impact on road planning!
I was stuck in traffic today for a few hours on the way to New Scientist Live. Missed loads of it. My bad planning. Worst luck. Really...
Don't touch my *!$@%&* board
I had a great time last weekend with a garden party with my friend and one of my longstanding tech leads David. It was great to spend the...
The negative externality of badly written SDKs
I've taken the last week off. For the first time in about a decade I haven't responded to emails (at least I've picked and chosen), Teams...
Data Smoosh
I was mulling over whether to derive a new jocular term for a Data Mesh. I pondered Data Mess but that seemed to obvious so I've opted...
The Loser CTO Cycle
I felt compelled to write about this because it's a phenomenon I'm seeing more and more. Cloud and data is revolutionary because it's...
Dropping a SQL table in your Synapse Spark notebooks (Python Edition)
So since writing the original post about how to drop a SQL table from a Spark Notebook I've been meaning to follow it up with a version...
Dropping a SQL table in your Synapse Spark notebooks
For the Python version of the code below, see the follow-up post. One of the nice things with Spark Pools in Azure Synapse Analytics is...
What makes a data engineer?
Note: this post is mainly about Azure but it can apply to any cloud. You get to a stage when you hear enough definitions of what people...
Documentation the easy way
So a slight departure from Spark (sort of) for this post, but I wanted to look at one of the most commonly overlooked aspects of building...
Processing Excel Data using Spark with Azure Synapse Analytics
Having recently released the Excel data source for Spark 3, I wanted to follow up with a "lets use it to process some Excel data" post....
Using Spark to read from Excel
People have data in Excel, so lets have a look at how we can read that data using Spark
Just one more column, what could go wrong?
Sometimes, when you go scanning through the documentation for Spark, you come across notes about certain functions. These tend to offer...
Why leave bad data to chance?
Something that we often see as Spark jobs are moved into production is that handling of bad data is either ignored, or a lot of effort...
Pivot, Step, Pivot, Twist, Un-pivot
Getting data into a good shape is a key part to Data Engineering, and we often get data in all sorts of shape and quality
When in doubt, shell out
The command line is a powerful environment that lets you do a lot of work quickly, easily, and in a repeatable way