Information & Data Management

Data Preparation: Why 80 Percent of AI Success Lies in Data Preparation

Data Preparation determines AI success or failure. Learn why 80 percent of the effort lies in data preparation and how SMEs can master it.

Author
SIMO GmbH
Published on
Reading time
12 minutes

The enthusiasm for artificial intelligence in the German mid-market is greater than ever. Companies are investing in foundation models, training chatbots, and experimenting with AI agents. Yet an inconvenient truth is regularly overlooked: The actual success factor lies not in the model but in the data that feeds it. Experience shows that approximately 80 percent of the total effort in a data science project goes into data preparation—that is, the work that takes place before the actual analysis and training. This figure is not an exaggeration but an industry-wide recognized empirical value that remains unchanged in 2025 and 2026. Those who underestimate this effort risk failed AI projects, wasted budgets, and missed competitive opportunities. This article explains why Data Preparation is so critical, which concrete steps are necessary, and how small and medium-sized enterprises can master the process systematically.

The Problem: Why So Many AI Projects Fail Because of Data

The statistics are sobering. According to a current Gartner forecast, organizations will abandon approximately 60 percent of their AI projects by the end of 2026 because the underlying data is not AI-ready. At the same time, the AI Study 2025 by maximal.digital shows that 83 percent of SMEs in Germany have no comprehensive data strategy. This means: The vast majority of mid-sized companies are launching AI initiatives on a foundation that cannot withstand the demands.

The 80/20 Rule of Data Science

The so-called 80/20 rule is one of the most frequently cited empirical values in data science. Data scientists spend up to 80 percent of their working time on data preprocessing—that is, collecting, cleaning, transforming, and validating data. Only 20 percent of time remains for the actual analysis, model training, and deriving insights. Computerwoche confirms this ratio and emphasizes that the process is exploratory and iterative: What is helpful for one dataset does not necessarily make sense for other data. Different methods must therefore be tested empirically.

This reality stands in sharp contrast to public perception. In media coverage, powerful models, impressive benchmarks, and spectacular use cases dominate. The laborious work of data preparation receives hardly any attention—even though it accounts for the lion’s share of success.

What Data Preprocessing Concretely Involves

Data Preparation is not a single work step but a multi-stage process with numerous sub-tasks. The following table provides an overview of the most important phases, their typical time share, and the most common challenges:

  • Phase · Description · Typical Time Share · Most Common Challenges
  • Data collection · Identification and consolidation of relevant data sources · 15 to 20 percent · Data silos, different formats, missing interfaces
  • Data cleaning · Detection and correction of errors, duplicates, and inconsistencies · 25 to 30 percent · Manual entry errors (average error rate 4.8 percent), outdated entries
  • Data transformation · Conversion to uniform formats, normalization, feature engineering · 15 to 20 percent · Different units of measurement, date formats, encodings
  • Data enrichment · Supplementing missing values, merging with external sources · 10 to 15 percent · Missing reference data, licensing costs of external data sources
  • Data validation · Quality checks, plausibility checks, documentation · 10 to 15 percent · Missing quality criteria, lack of test data

The Fraunhofer Academy points out that this process rarely runs linearly in practice. When poor analysis results can be traced back to the data, data preparation must be repeated and the entire analysis process run through again. This iterative approach explains why the time investment is so high.

Core Content: Data Preparation as a Strategic Success Factor

The Economic Costs of Poor Data

The financial impacts of inadequate data quality are considerable. According to Gartner, poor data quality costs companies an average of 12.9 million US dollars per year. A study by the IBM Institute for Business Value shows that over a quarter of surveyed organizations record losses of more than 5 million US dollars annually due to poor data quality. In the United States alone, the costs of bad data total over 3.1 trillion US dollars per year according to IBM.

For the German mid-market, these figures may initially seem abstract. Yet the relative impacts are comparable: Incorrect inventory levels lead to wrong deliveries, duplicate customer records distort revenue forecasts, outdated supplier information causes procurement problems. MIT Sloan Management Review puts the annual revenue loss from poor data quality at 15 to 25 percent—a magnitude that no company can ignore.

Industry Example: Data Preparation in Mid-Market Manufacturing

A hands-on example illustrates the scale. A mid-sized machine manufacturer with 280 employees and annual revenue of 45 million euros wanted to introduce AI-powered predictive maintenance. The goal: Reduce unplanned machine downtimes by 30 percent and lower maintenance costs by 15 percent.

The starting situation was typical for the mid-market: Sensor data existed in three different formats because the machines came from different manufacturers. Maintenance logs existed partly digitally in the ERP system, partly as handwritten entries. Spare parts data was spread across four separate spreadsheets. Historical downtime records were only sporadically documented.

The result: Of the planned twelve project months, nine months—75 percent—were spent on data preparation. Specifically, this included: Unification of sensor data formats (8 weeks), digitization and structuring of maintenance logs (10 weeks), consolidation of spare parts data in a central database (6 weeks), cleaning and supplementing of downtime documentation (8 weeks), and validation and quality assurance of the entire dataset (4 weeks).

Only after this foundational work was completed could the actual AI model be trained. The training itself took only six weeks. The result after one year of operation: 28 percent fewer unplanned downtimes and 12 percent lower maintenance costs. Without thorough data preparation, the model would have failed on inconsistent and incomplete data.

Why the Mid-Market Is Particularly Affected

The AI Study 2025 by maximal.digital paints a clear picture of the challenges. 76 percent of SMEs struggle with inadequate data quality and data silos. Only 23 percent have successfully implemented concrete AI projects. 68 percent of surveyed companies have no developed AI roadmap, and only 19 percent have established a dedicated AI responsible person or AI team.

Large corporations address these challenges with specialized data engineering teams, Chief Data Officers, and multi-person data governance departments. SMEs typically do not have these resources. Data is maintained in different departments, systems, and formats—often without central responsibility and without binding standards. At the same time, regulatory pressure is increasing: The EU AI Act will become fully effective in 2026 and places strict requirements on documentation, risk management, and data quality, particularly for high-risk AI systems.

The Future of Data Preparation: AI as a Tool for Better Data

A promising approach is using AI itself to improve data preparation. According to a Gartner forecast, by 2027 approximately 75 percent of analytics content will be contextualized and enriched by generative AI. The Data Economist sees the potential to reverse the traditional 80/20 ratio: AI-powered automation could reduce the effort for data preparation to 20 percent and free 80 percent of capacity for actual analysis and decision-making.

However, the Fraunhofer Academy warns against excessive expectations. The complete automation of data preparation is the “Holy Grail” of data science, but due to the complexity of various quality aspects, contexts, and analysis algorithms, it has not yet been achieved. The success of data preparation continues to depend heavily on the skills of the involved data experts.

For SMEs, this means: AI-powered data preparation tools can significantly accelerate routine tasks such as duplicate cleaning, format standardization, and anomaly detection. But they do not replace strategic planning, the definition of quality criteria, and the professional assessment of data relevance.

Practical Guide: Mastering Data Preparation in Five Steps

The following guide is specifically aimed at small and medium-sized enterprises that want to systematically build their data preparation. It is based on best practices that Mittelstand-Digital, Computerwoche, and the KIWise Whitepaper 2025 consistently recommend.

Step 1: Conduct a Data Inventory

Before you can clean or transform data, you need to know what data actually exists. Conduct a complete inventory of all data sources. For each source, capture the format, update cycle, responsibility, and current quality assessment. Use structured workshops with all relevant business departments for this purpose.

Practical tip: Start with the core processes of your company. In a manufacturing operation, these are typically order processing, production, warehousing, and financial accounting. For each of these domains, you should identify and document the primary data sources.

Step 2: Assess and Clean Data Quality

Conduct a systematic readiness check that assesses the maturity of your data across four dimensions: Completeness (are relevant data points missing?), Correctness (do the values match reality?), Consistency (do different sources deliver the same values for the same facts?), and Currency (are the data up to date?).

Prioritize cleaning by business benefit. Not all data needs to be perfect immediately. Focus on the data areas most relevant to your planned AI use case.

Step 3: Establish Standards and Governance

Define binding rules for data entry and maintenance. This includes naming conventions, mandatory fields, permitted value lists, and format specifications. Designate a Data Owner for each data area who is responsible for quality.

This step is critical because it prevents cleaned data from immediately becoming contaminated again. Without governance, every cleanup is a fight against windmills—data quality will revert to its previous level after a short time.

Step 4: Automate Data Pipelines

Only when the foundations are in place does investment in automation pay off. Replace manual data transfers with automated interfaces and pipelines. Use ETL tools (Extract, Transform, Load) or workflow automation platforms like n8n to standardize recurring data processing steps.

Important: Only automate processes whose data quality is already secured. Otherwise you automate the error—faulty data flows faster and more reliably through the system, but it remains faulty.

Step 5: Monitoring and Continuous Improvement

Data Preparation is not a one-time project but an ongoing process. Implement dashboards that display key data quality KPIs in real time. Define thresholds and alerts that automatically raise the alarm when quality deviates. Conduct quarterly data quality reviews.

The Technovice forecast for 2026 confirms this approach: Data preparation is increasingly understood as a continuous system encompassing ingestion, labeling, enrichment, transformation, versioning, and access control. Every dataset must be traceably documented.

Frequently Asked Questions (FAQs)

Is it really true that 80 percent of the effort goes into Data Preparation?

Yes, this empirical value is confirmed by numerous sources including Computerwoche, the Fraunhofer Academy, and the portal ki.engineering. The exact percentage varies by project and data quality between 60 and 80 percent, but the order of magnitude is recognized industry-wide. Forbes reports that approximately 60 percentage points of this fall on cleaning and organizing the data alone.

What does poor data quality concretely cost a mid-sized company?

The costs depend on company size and industry. According to MIT Sloan Management Review, companies lose 15 to 25 percent of their annual revenue to poor data quality. For an SME with 10 million euros in revenue, this means potential losses of 1.5 to 2.5 million euros annually—through wrong deliveries, manual rework, missed opportunities, and wrong decisions.

Can AI tools not automate data preparation?

Partially. AI-powered data preparation tools can significantly accelerate routine tasks such as duplicate cleaning, format recognition, and anomaly detection. The Fraunhofer Academy warns, however, that complete automation is not yet possible due to the complexity of various quality aspects and contexts. Human expertise remains indispensable for strategic planning and professional assessment.

How long does thorough data preparation for an AI project take?

This depends heavily on the initial state of the data and the complexity of the use case. For a typical mid-market AI project, you should plan for 3 to 9 months. The industry example of the machine manufacturer shows that 75 percent of the total project duration can be spent on data preparation. Companies that already have good data governance in place can significantly shorten this timeframe.

What role does the EU AI Act play in data preparation?

The EU AI Act will become fully effective from August 2026 and places strict requirements on data quality, documentation, and traceability—particularly for high-risk AI systems. GDPR, EU AI Act, and the Data Governance Act require proof of data origin and documentation of training data. Companies that do not systematically document their data preparation risk not only technical failure but also regulatory consequences.

References

The following sources form the basis of this article and were published or evaluated in the period from 2025 to March 2026:

Tags

  • SMEs
  • Data Preparation
  • Data Quality
  • Best Practices
  • Mid-Market

Back to the overview

Business Data Strategy for your company

From the target state to Delivery Supervision. We advise you and enable your organization.

Book a discovery callView our advisory work