←back to thread

Ask HN: How are you cleaning and transforming data before imports/uploads?

36 points dataflowmapper | 1 comments | 14 May 25 16:27 UTC | HN request time: 0.228s | source

Hi all,

I’m curious how folks handle the prep work for data imports/uploads into systems like Salesforce, Workday, NetSuite, or really any app that uses template based import for data loading, migration, or implementation.

Specifically: - How do you manage conversions/transformations like formatting dates, getting everything aligned with the templates, mapping old codes to new ones, etc.

- Are you primarily using Excel, custom scripts, Power Query or something else?

- What are the most tedious/painful parts of this process and what have you found that works?

Really appreciate any insights and am curious to learn from everyone's experience.

Show context
PaulHoule ◴[14 May 25 16:34 UTC] No.43986506[source]▶
"Scripts" in Python, Java and other conventional programming languages (e.g. whatever it is you already use)

Not Bash, not Excel, not any special-purpose tool because the motto of those is "you can't get there from here". Maybe you can get 80% of the way there, which is really seductive, but that last 20% is like going to the moon. Specifically, real programming languages have the tools to format dates correctly with a few lines of code you can wrap into a function, fake programming languages don't. Mapping codes is straightforward, etc.

replies(3): >>43986637 #>>43987179 #>>43987624 #
chaos_emergent ◴[14 May 25 16:46 UTC] No.43986637[source]▶
for the longest time I envisioned some sort of configuration specification that could retrieve URLs, transform and map data, handle complex conditional flows...and then I realized that I wanted a Normal Programming Language for Commerce and started asking o3 to write me Python scripts.
replies(1): >>43986929 #
1. aaronbrethorst ◴[14 May 25 17:13 UTC] No.43986929[source]▶
Hell, for me, would be what you described and implemented in Yaml.