Cleaning data is essential to making sure that data science projects are executed with the highest level of accuracy possible. Manual cleaning calls for extensive work, though, and it also can induce human errors along the way. For this reason, automated solutions, often based on basic statistical models, are used to eliminate flawed entries. It's a good idea, though, to develop some understanding of the top strategies for dealing with the job.
A lot of undesirable data can be cleaned up using common pattern-matching techniques. The standard tool for the job is usually a programming language that handles regular expressions well. Done right, a single line of code should serve the purpose well.
1) Cleaning out and fixing characters is almost always the first step in data cleaning. This usually entails removing unnecessary spaces, HTML entity characters and other elements that might interfere with machine or human reading. Many languages and spreadsheet applications have TRIM functions that can rapidly eliminate bad spaces, and regular expressions and built-in functions usually will do the rest.
2) Duplicate removal is a little trickier because it's critical to make sure you're only removing true duplicates. Using other good data management techniques will make duplicate removal simpler, such as indexing. Near-duplicates, though, can be tricky, especially if the original data entry was performed sloppily.
While we tend to think of data cleaning as mostly preparing information for use, it also is helpful in improving efficiency. Storage and processing efficiency are both ripe areas for improvement.
3) Converting fields makes a big difference sometimes to storage. If you've imported numerical fields, for example, and they all appear in text columns, you'll likely benefit from turning those columns into integers, decimals or floats.
4) Reducing processing overhead is also a good choice. A project may only require a certain level of decimal precision, and rounding off numbers and storing them in smaller memory spaces can speed things up significantly. Just make sure you're not kneecapping required decimal precision when you use this approach.
Folks in the stats world have been trying to find ways to improve data quality for decades. Many of their techniques are ideal for data cleaning, too.
5) Outlier removal and the use of limits are common ways to analyze a dataset and determine what doesn't belong. By analyzing a dataset for extreme and rare data points, you can quickly pick out what might be questionable data. Be careful, though, to recheck your data afterward to verify that low-quality data was removed rather than data about exceptional outcomes.
Limiting factors also make for excellent filters. If you know it's impossible for an entry to register a zero, for example, installing a limit above that mark can eliminate times when a data source simply returned a blank.
6) Validation models are useful for verifying that your data hasn't been messed up by all the manipulation. If you see validation numbers that scream that something has gone wrong, you can go back through your data cleaning process to identify what might have misfired.
