top of page
Pies, Lies and AIs
Showing the struggle of organisations to address the data revolution
Pies, Lies and AIs
Exploring the world of data and organisational intransigence
Search
dazfuller
Dropping a SQL table in your Synapse Spark notebooks (Python Edition)
So since writing the original post about how to drop a SQL table from a Spark Notebook I've been meaning to follow it up with a version...
dazfuller
Dropping a SQL table in your Synapse Spark notebooks
For the Python version of the code below, see the follow-up post. One of the nice things with Spark Pools in Azure Synapse Analytics is...
dazfuller
Documentation the easy way
So a slight departure from Spark (sort of) for this post, but I wanted to look at one of the most commonly overlooked aspects of building...
dazfuller
Processing Excel Data using Spark with Azure Synapse Analytics
Having recently released the Excel data source for Spark 3, I wanted to follow up with a "lets use it to process some Excel data" post....
dazfuller
Using Spark to read from Excel
People have data in Excel, so lets have a look at how we can read that data using Spark
dazfuller
Just one more column, what could go wrong?
Sometimes, when you go scanning through the documentation for Spark, you come across notes about certain functions. These tend to offer...
dazfuller
Why leave bad data to chance?
Something that we often see as Spark jobs are moved into production is that handling of bad data is either ignored, or a lot of effort...
dazfuller
Pivot, Step, Pivot, Twist, Un-pivot
Getting data into a good shape is a key part to Data Engineering, and we often get data in all sorts of shape and quality
dazfuller
When in doubt, shell out
The command line is a powerful environment that lets you do a lot of work quickly, easily, and in a repeatable way
Home: Blog2
Home: Subscribe
CONTACT
Home: Contact
bottom of page