What is Data Science? The Field That Turned Raw Data Into Billion-Dollar Decisions
Data science is the discipline of extracting meaning from data using statistics, programming, and domain knowledge. It is also one of the most poorly defined job titles in the industry. Here is what the field actually covers, what the tooling looks like, and what real data science work involves.
Every major technology company runs on data science. Spotify's recommendations that surface music you did not know you wanted. Netflix's prediction of which shows you will actually finish. Amazon's pricing algorithms that adjust millions of product prices daily based on demand signals. Google's ad targeting that turns search intent into revenue. The fraud detection that blocked your card when someone tried to use it in a city you have never visited.
All of that is data science. Not magic. Not a black box of mysterious AI. Statistics, mathematics, and programming applied to data to extract patterns and drive decisions.
The field has a definition problem. Data scientist was named "the sexiest job of the 21st century" by Harvard Business Review in 2012 and the title has been attached to wildly different roles ever since. Some companies call their SQL analysts data scientists. Some call their machine learning engineers data scientists. Some have separate titles for data engineer, data analyst, ML engineer, research scientist, and applied scientist, and then call all of them data science anyway.
This article cuts through the noise. It covers what data science actually is, what the different roles in the broader field do, what the tooling looks like, and what real data science work involves with production-quality code examples throughout.
The Core Definition
Data science is the discipline of extracting actionable insights from data using a combination of statistics, programming, and domain expertise. That three-part combination is important. Statistics without programming cannot handle modern data volumes. Programming without statistics produces systems that compute wrong answers efficiently. Domain expertise without both is guesswork dressed up in charts.
The term "data science" was popularized around 2008-2012. The underlying work is older. Statistical computing, machine learning research, and business intelligence have existed for decades. What changed was the volume of data companies were collecting, the decrease in storage and compute costs, and the maturation of Python's scientific computing ecosystem into something usable by practitioners who are not PhD researchers.
The field sits at the intersection of three domains:
Mathematics and Statistics provides the theoretical foundation: probability, linear algebra, calculus, statistical inference, hypothesis testing, regression, dimensionality reduction. You do not need to derive everything from first principles, but you need to understand what the tools are doing mathematically to know when they are lying to you.
Computer Science and Programming provides the implementation capability. SQL for querying databases. Python for data manipulation, modeling, and automation. Distributed computing frameworks for data that does not fit on one machine. Software engineering practices for deploying models to production.
Domain Knowledge is what separates a data scientist who produces useful work from one who produces technically correct but practically irrelevant results. A model predicting customer churn that ignores the fact that your contract renewal cycle happens every January will underperform a simpler model built by someone who understands the business.
The Roles Inside Data Science
The field has fragmented into distinct specializations. Understanding the distinctions helps you understand which role you are pursuing and what skills matter.
Data Analyst focuses on describing what happened. They write SQL, build dashboards, create reports, and answer specific business questions. "What was our conversion rate last month?" "Which marketing channel drove the most revenue?" "Did the new feature increase engagement?" Analysts work primarily in SQL, Excel, and BI tools like Tableau, Looker, or Power BI. They are the first line of data work in most organizations.
Data Scientist focuses on predicting what will happen and prescribing what to do. They build statistical models and machine learning models to identify patterns, make predictions, and quantify uncertainty. "Which customers are likely to churn in the next 30 days?" "What price maximizes expected revenue for this product?" "Which users should we target with this promotion?" Data scientists work primarily in Python, using the scientific computing stack.
Machine Learning Engineer focuses on deploying models to production and keeping them running. They bridge the gap between a data scientist's Jupyter notebook and a model serving 10 million requests per day. MLEs understand model serving infrastructure, feature stores, A/B testing frameworks, model monitoring, and retraining pipelines. They write production-grade software.
Data Engineer builds and maintains the infrastructure that data flows through. They build ETL (Extract, Transform, Load) pipelines, data warehouses, streaming data systems, and the ingestion layer that gets raw data from source systems into formats analysts and scientists can work with. Data engineers work primarily with SQL, Spark, Airflow, Kafka, and cloud data services. Their outputs are the reliable data foundations that everyone else depends on.
Research Scientist works on novel methods and algorithms, typically at large tech companies with dedicated research functions. They publish papers, develop new model architectures, and explore techniques that may or may not make it into production. This is the most academically oriented role.
Most teams blend these roles. A small company might have one person doing all of them. A larger team might have a data engineer keeping pipelines running, analysts answering business questions, a data scientist building models, and an ML engineer deploying them. Understanding which role you are filling at any given moment helps you prioritize correctly.
The Python Data Science Stack
Python owns data science. This is a factual statement about the current state of the ecosystem, not an endorsement. R has a devoted following in academic statistics and biostatistics. Julia has advocates for high-performance numerical computing. MATLAB is entrenched in engineering. But for data science as practiced in the technology industry, Python is the primary language and has been for almost a decade.
The reason is the ecosystem: NumPy, Pandas, Matplotlib, Scikit-learn, PyTorch, TensorFlow, Jupyter, and the hundreds of libraries built on top of them. Everything interoperates through a common data model based on NumPy arrays. A NumPy array from your data loading code passes directly into your Scikit-learn model, into your visualization library, into your feature engineering code. The ecosystem is cohesive in a way that took years to develop and cannot be replicated quickly.
Exploratory Data Analysis
Real data is messy. Before you build any models, you need to understand the structure, distributions, and quality of your data. Exploratory Data Analysis (EDA) is the systematic process of doing that.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
pd.set_option('display.max_columns', 50)
pd.set_option('display.float_format', '{:.4f}'.format)
np.random.seed(42)
n = 50000
df = pd.DataFrame({
'customer_id': range(1, n + 1),
'age': np.random.normal(38, 12, n).clip(18, 80).astype(int),
'tenure_months': np.random.exponential(24, n).clip(1, 120).astype(int),
'monthly_charges': np.random.gamma(3, 15, n).round(2),
'support_tickets_90d': np.random.poisson(1.5, n),
'login_frequency': np.random.exponential(8, n).clip(0, 50).round(1),
'contract_type': np.random.choice(
['month-to-month', 'one-year', 'two-year'],
n,
p=[0.5, 0.3, 0.2]
),
'payment_method': np.random.choice(
['credit_card', 'bank_transfer', 'check', 'digital_wallet'],
n,
p=[0.35, 0.30, 0.15, 0.20]
),
'has_tech_support': np.random.choice([0, 1], n, p=[0.4, 0.6]),
'has_streaming': np.random.choice([0, 1], n, p=[0.45, 0.55]),
})
churn_prob = (
0.15
+ 0.30 * (df['contract_type'] == 'month-to-month').astype(float)
- 0.12 * (df['contract_type'] == 'two-year').astype(float)
+ 0.15 * (df['support_tickets_90d'] > 3).astype(float)
+ 0.10 * (df['tenure_months'] < 6).astype(float)
- 0.08 * (df['tenure_months'] > 48).astype(float)
- 0.05 * df['has_tech_support'].astype(float)
+ 0.08 * (df['login_frequency'] < 2).astype(float)
).clip(0.02, 0.95)
df['churned'] = (np.random.random(n) < churn_prob).astype(int)
df['monthly_charges'] = df['monthly_charges'] + np.where(df['has_streaming'], 15, 0)
df.loc[np.random.choice(df.index, 200), 'monthly_charges'] = np.nan
df.loc[np.random.choice(df.index, 50), 'age'] = -1
print("=== DATASET OVERVIEW ===")
print(f"Shape: {df.shape}")
print(f"\nData types:\n{df.dtypes}")
print(f"\nMissing values:\n{df.isnull().sum()}")
print(f"\nDuplicate rows: {df.duplicated().sum()}")
print("\n=== NUMERICAL SUMMARY ===")
print(df.describe().T)
print("\n=== CATEGORICAL SUMMARY ===")
for col in ['contract_type', 'payment_method']:
print(f"\n{col}:")
print(df[col].value_counts(normalize=True).round(3))
print(f"\n=== TARGET VARIABLE ===")
print(f"Churn rate: {df['churned'].mean():.1%}")
print(f"Class imbalance ratio: {(1 - df['churned'].mean()) / df['churned'].mean():.1f}:1")
print("\n=== CHURN BY SEGMENT ===")
for col in ['contract_type', 'payment_method', 'has_tech_support', 'has_streaming']:
segment_churn = df.groupby(col)['churned'].agg(['mean', 'count']).round(3)
segment_churn.columns = ['churn_rate', 'count']
print(f"\n{col}:\n{segment_churn}")
numerical_cols = ['age', 'tenure_months', 'monthly_charges', 'support_tickets_90d', 'login_frequency']
correlations = df[numerical_cols + ['churned']].corr()['churned'].drop('churned').sort_values()
print(f"\n=== CORRELATIONS WITH CHURN ===\n{correlations}")
data_quality_issues = []
if (df['age'] < 0).any():
data_quality_issues.append(f"Negative age values: {(df['age'] < 0).sum()}")
if df['monthly_charges'].isnull().any():
data_quality_issues.append(f"Missing monthly_charges: {df['monthly_charges'].isnull().sum()}")
if (df['monthly_charges'] < 0).any():
data_quality_issues.append(f"Negative charges: {(df['monthly_charges'] < 0).sum()}")
print(f"\n=== DATA QUALITY ISSUES ===")
for issue in data_quality_issues:
print(f" - {issue}")
The EDA phase is where you find the problems that will break your models. Negative ages are not a valid input to a model. Missing values need a strategy: impute them, drop rows, or use algorithms that handle missing data natively. Understanding the class imbalance (15% churn rate means 85% of rows are class 0) tells you that a model that always predicts "no churn" would be 85% accurate but completely useless.
The correlation analysis surfaces which features have predictive signal. Negative correlation between tenure_months and churn makes intuitive sense: customers who have been with you for four years are less likely to leave than customers who signed up last month. That intuition needs to be verified in the data, and verifying it gives you confidence in your feature engineering decisions.
Feature Engineering
Feature engineering is the craft of transforming raw data into representations that models can learn from effectively. Most of the performance gains in applied machine learning come from better features, not better algorithms.
from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
import warnings
warnings.filterwarnings('ignore')
def clean_and_engineer_features(df: pd.DataFrame) -> pd.DataFrame:
df = df.copy()
df['age'] = df['age'].where(df['age'] >= 0, np.nan)
df['monthly_charges'] = df['monthly_charges'].where(df['monthly_charges'] >= 0, np.nan)
df['log_tenure'] = np.log1p(df['tenure_months'])
df['tenure_bucket'] = pd.cut(
df['tenure_months'],
bins=[0, 6, 12, 24, 48, 120],
labels=['new', 'short', 'medium', 'long', 'veteran']
)
df['charge_per_month'] = df['monthly_charges'].fillna(df['monthly_charges'].median())
df['is_high_value'] = (df['monthly_charges'] > df['monthly_charges'].quantile(0.75)).astype(int)
df['support_intensity'] = df['support_tickets_90d'] / (df['tenure_months'] / 3 + 1)
df['is_low_engagement'] = (df['login_frequency'] < 2).astype(int)
df['is_high_support'] = (df['support_tickets_90d'] > 3).astype(int)
df['has_multiple_services'] = (df['has_tech_support'] + df['has_streaming'] >= 2).astype(int)
df['is_month_to_month'] = (df['contract_type'] == 'month-to-month').astype(int)
df['is_two_year'] = (df['contract_type'] == 'two-year').astype(int)
df['uses_auto_payment'] = (df['payment_method'].isin(['credit_card', 'bank_transfer', 'digital_wallet'])).astype(int)
df['risk_score'] = (
df['is_month_to_month'] * 2
+ df['is_high_support']
+ df['is_low_engagement']
- df['is_two_year']
- df['uses_auto_payment'] * 0.5
- df['has_multiple_services'] * 0.5
)
return df
df_engineered = clean_and_engineer_features(df)
numerical_features = [
'age', 'tenure_months', 'log_tenure', 'monthly_charges',
'support_tickets_90d', 'login_frequency', 'support_intensity',
'charge_per_month', 'risk_score'
]
categorical_features = ['contract_type', 'payment_method', 'tenure_bucket']
binary_features = [
'has_tech_support', 'has_streaming', 'is_high_value',
'is_low_engagement', 'is_high_support', 'has_multiple_services',
'is_month_to_month', 'is_two_year', 'uses_auto_payment'
]
numerical_transformer = Pipeline([
('imputer', KNNImputer(n_neighbors=5)),
('scaler', StandardScaler())
])
categorical_transformer = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('encoder', OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore'))
])
preprocessor = ColumnTransformer([
('numerical', numerical_transformer, numerical_features),
('categorical', categorical_transformer, categorical_features),
('binary', 'passthrough', binary_features)
])
The engineered features capture business logic that raw numbers cannot. support_intensity normalizes support ticket counts by tenure: a customer who filed 3 tickets in their first month is a different signal than one who filed 3 tickets across five years. risk_score combines multiple risk factors into a single number that represents accumulated behavioral risk. These features encode domain knowledge in a form the model can use.
The ColumnTransformer applies different preprocessing strategies to different column types. Numerical features get KNN imputation (uses the k nearest neighbors to fill missing values) and standardization (zero mean, unit variance). Categorical features get mode imputation and one-hot encoding. Binary features pass through without transformation. This whole preprocessing chain is a single Scikit-learn transformer that fits on training data and applies to new data at inference time.
Model Building and Evaluation
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate
from sklearn.ensemble import (
RandomForestClassifier, GradientBoostingClassifier,
VotingClassifier, StackingClassifier
)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
roc_auc_score, average_precision_score, classification_report,
confusion_matrix, RocCurveDisplay, PrecisionRecallDisplay
)
from sklearn.calibration import CalibratedClassifierCV
import xgboost as xgb
feature_columns = numerical_features + categorical_features + binary_features
X = df_engineered[feature_columns].copy()
y = df_engineered['churned']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
scale_pos_weight = (y_train == 0).sum() / (y_train == 1).sum()
models = {
'Logistic Regression': Pipeline([
('preprocessor', preprocessor),
('classifier', LogisticRegression(
C=0.1, class_weight='balanced', max_iter=1000, random_state=42
))
]),
'Random Forest': Pipeline([
('preprocessor', preprocessor),
('classifier', RandomForestClassifier(
n_estimators=300, max_depth=8, min_samples_leaf=20,
class_weight='balanced', n_jobs=-1, random_state=42
))
]),
'XGBoost': Pipeline([
('preprocessor', preprocessor),
('classifier', xgb.XGBClassifier(
n_estimators=300, max_depth=6, learning_rate=0.05,
subsample=0.8, colsample_bytree=0.8,
scale_pos_weight=scale_pos_weight, eval_metric='auc',
use_label_encoder=False, random_state=42, n_jobs=-1
))
]),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}
for name, pipeline in models.items():
cv_results = cross_validate(
pipeline, X_train, y_train, cv=cv,
scoring=['roc_auc', 'average_precision'],
return_train_score=True, n_jobs=-1
)
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
y_prob = pipeline.predict_proba(X_test)[:, 1]
results[name] = {
'cv_auc_mean': cv_results['test_roc_auc'].mean(),
'cv_auc_std': cv_results['test_roc_auc'].std(),
'cv_ap_mean': cv_results['test_average_precision'].mean(),
'test_auc': roc_auc_score(y_test, y_prob),
'test_ap': average_precision_score(y_test, y_prob),
'pipeline': pipeline
}
print(f"\n{'='*50}")
print(f"Model: {name}")
print(f"CV AUC: {results[name]['cv_auc_mean']:.4f} ± {results[name]['cv_auc_std']:.4f}")
print(f"Test AUC: {results[name]['test_auc']:.4f}")
print(f"Test AP: {results[name]['test_ap']:.4f}")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=['Retained', 'Churned']))
best_model_name = max(results, key=lambda x: results[x]['test_auc'])
best_pipeline = results[best_model_name]['pipeline']
print(f"\nBest model: {best_model_name} (AUC: {results[best_model_name]['test_auc']:.4f})")
calibrated_model = CalibratedClassifierCV(best_pipeline, cv=3, method='isotonic')
calibrated_model.fit(X_train, y_train)
AUC (Area Under the ROC Curve) measures how well your model discriminates between classes across all possible classification thresholds. An AUC of 0.5 is random. 0.7 is acceptable for a difficult problem. 0.85+ is strong. Average Precision (the area under the Precision-Recall curve) is a better metric when classes are imbalanced, which they almost always are in churn prediction.
Cross-validation with 5 folds gives a more honest performance estimate than a single train/test split. You train on 80% of the training data and validate on 20%, five times, rotating which 20% is held out. The mean and standard deviation of the validation scores tell you both how good the model is and how stable that estimate is. High variance across folds means your model is sensitive to which data it trains on.
class_weight='balanced' and scale_pos_weight address class imbalance by giving the minority class (churned customers) proportionally higher weight in the loss function. Without this, models tend to predict the majority class (retained) for most inputs because that maximizes raw accuracy.
Calibration matters when you use model probabilities to make decisions. A model that outputs 0.7 for "will churn" should be right 70% of the time on average. Uncalibrated models often output probabilities that are systematically too high or too low. Isotonic regression calibration adjusts the probability outputs to match observed frequencies.
Feature Importance and Explainability
import shap
xgb_pipeline = results['XGBoost']['pipeline']
X_train_transformed = xgb_pipeline.named_steps['preprocessor'].transform(X_train)
cat_encoder = xgb_pipeline.named_steps['preprocessor'].transformers_[1][1].named_steps['encoder']
cat_feature_names = cat_encoder.get_feature_names_out(categorical_features).tolist()
transformed_feature_names = numerical_features + cat_feature_names + binary_features
xgb_model = xgb_pipeline.named_steps['classifier']
explainer = shap.TreeExplainer(xgb_model)
shap_values = explainer.shap_values(X_train_transformed[:1000])
shap_df = pd.DataFrame(
np.abs(shap_values),
columns=transformed_feature_names
)
feature_importance = shap_df.mean().sort_values(ascending=False).head(15)
print("Top 15 features by mean |SHAP value|:")
print(feature_importance.to_string())
sample_idx = 0
sample_shap = dict(zip(transformed_feature_names, shap_values[sample_idx]))
top_factors = sorted(sample_shap.items(), key=lambda x: abs(x[1]), reverse=True)[:5]
print(f"\nTop churn factors for sample customer:")
for feature, shap_val in top_factors:
direction = "increases" if shap_val > 0 else "decreases"
print(f" {feature}: {shap_val:.3f} ({direction} churn probability)")
SHAP (SHapley Additive exPlanations) values explain individual model predictions by computing each feature's contribution to the prediction. The mean absolute SHAP value across the dataset is a model-agnostic feature importance measure that accounts for feature interactions.
This matters in production. A model that predicts "customer will churn with 82% probability" is useful. A model that says "customer will churn with 82% probability because they have filed 5 support tickets in the last 90 days, their contract is month-to-month, and they have only logged in twice in the last month" is actionable. The customer success team can call that customer and offer them a long-term contract discount before they leave.
Data Engineering: The Foundation
Data scientists work with clean, processed data. Data engineers produce that data. The pipeline from raw production data to something usable for analysis is an engineering problem that consumes the majority of most data teams' time.
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from typing import Optional
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
class DataPipeline:
def __init__(self, source_db_url: str, dest_db_url: str):
self.source_db_url = source_db_url
self.dest_db_url = dest_db_url
self.processing_date = datetime.now()
def extract(self, lookback_days: int = 90) -> pd.DataFrame:
cutoff = self.processing_date - timedelta(days=lookback_days)
query = """
SELECT
c.id AS customer_id,
c.created_at AS customer_since,
c.contract_type,
c.payment_method,
c.monthly_charges,
c.has_tech_support,
c.has_streaming,
EXTRACT(EPOCH FROM (NOW() - c.created_at)) / 2592000 AS tenure_months,
COUNT(DISTINCT st.id) FILTER (
WHERE st.created_at > %(cutoff)s
) AS support_tickets_90d,
COUNT(DISTINCT e.id) FILTER (
WHERE e.event_type = 'login'
AND e.created_at > %(cutoff)s
) AS logins_90d,
MAX(e.created_at) FILTER (
WHERE e.event_type = 'login'
) AS last_login_at,
c.churned_at IS NOT NULL AS churned,
c.churned_at
FROM customers c
LEFT JOIN support_tickets st ON st.customer_id = c.id
LEFT JOIN events e ON e.customer_id = c.id
WHERE c.created_at > NOW() - INTERVAL '3 years'
GROUP BY c.id, c.created_at, c.contract_type,
c.payment_method, c.monthly_charges,
c.has_tech_support, c.has_streaming, c.churned_at
"""
logger.info(f"Extracting data with cutoff: {cutoff}")
return pd.read_sql(query, self.source_db_url, params={'cutoff': cutoff})
def validate(self, df: pd.DataFrame) -> tuple[pd.DataFrame, dict]:
validation_report = {
'total_rows': len(df),
'issues': []
}
required_columns = [
'customer_id', 'customer_since', 'contract_type',
'monthly_charges', 'tenure_months', 'churned'
]
missing_cols = [c for c in required_columns if c not in df.columns]
if missing_cols:
raise ValueError(f"Missing required columns: {missing_cols}")
duplicates = df.duplicated(subset=['customer_id']).sum()
if duplicates > 0:
validation_report['issues'].append(f"Duplicate customer_ids: {duplicates}")
df = df.drop_duplicates(subset=['customer_id'], keep='last')
negative_charges = (df['monthly_charges'] < 0).sum()
if negative_charges > 0:
validation_report['issues'].append(f"Negative charges: {negative_charges}")
df = df[df['monthly_charges'] >= 0]
invalid_tenure = (df['tenure_months'] < 0).sum()
if invalid_tenure > 0:
validation_report['issues'].append(f"Negative tenure: {invalid_tenure}")
df = df[df['tenure_months'] >= 0]
missing_pct = df.isnull().mean() * 100
high_missing = missing_pct[missing_pct > 20]
if not high_missing.empty:
validation_report['issues'].append(
f"High missing rate columns: {high_missing.to_dict()}"
)
validation_report['rows_after_cleaning'] = len(df)
validation_report['rows_removed'] = validation_report['total_rows'] - len(df)
churn_rate = df['churned'].mean()
if not 0.01 <= churn_rate <= 0.50:
validation_report['issues'].append(
f"Unusual churn rate: {churn_rate:.1%}. Expected 1-50%."
)
logger.info(f"Validation complete: {validation_report}")
return df, validation_report
def transform(self, df: pd.DataFrame) -> pd.DataFrame:
df = df.copy()
df['login_frequency'] = df['logins_90d'] / 90
df['support_intensity'] = df['support_tickets_90d'] / (df['tenure_months'].clip(1) / 3)
df['tenure_bucket'] = pd.cut(
df['tenure_months'],
bins=[0, 6, 12, 24, 48, float('inf')],
labels=['new', 'short', 'medium', 'long', 'veteran']
)
df['days_since_login'] = (
(self.processing_date - pd.to_datetime(df['last_login_at']))
.dt.days
.fillna(999)
)
df['is_at_risk'] = (
(df['support_tickets_90d'] > 3) |
(df['days_since_login'] > 30) |
((df['contract_type'] == 'month-to-month') & (df['tenure_months'] < 12))
).astype(int)
df['processed_at'] = self.processing_date
df['pipeline_version'] = '2.1.0'
return df
def load(self, df: pd.DataFrame, table_name: str = 'customer_features') -> int:
from sqlalchemy import create_engine
engine = create_engine(self.dest_db_url)
df.to_sql(
table_name,
engine,
if_exists='replace',
index=False,
chunksize=10000,
method='multi'
)
logger.info(f"Loaded {len(df)} rows to {table_name}")
return len(df)
def run(self, lookback_days: int = 90) -> dict:
logger.info("Starting pipeline run")
start = datetime.now()
raw_df = self.extract(lookback_days)
logger.info(f"Extracted {len(raw_df)} rows")
clean_df, validation_report = self.validate(raw_df)
if len(clean_df) < 1000:
raise ValueError(f"Too few rows after cleaning: {len(clean_df)}")
feature_df = self.transform(clean_df)
rows_loaded = self.load(feature_df)
duration = (datetime.now() - start).total_seconds()
result = {
'status': 'success',
'rows_extracted': len(raw_df),
'rows_loaded': rows_loaded,
'duration_seconds': duration,
'validation_report': validation_report,
'processed_at': self.processing_date.isoformat()
}
logger.info(f"Pipeline complete: {result}")
return result
The pipeline follows the ETL pattern: extract raw data from the source, validate and clean it, transform it into features, load it into the destination. Each step is a method that does one thing. The validate method surfaces data quality issues without silently corrupting the dataset. The transform method encodes business logic. The load method handles the write.
Validation is where most pipelines fail when they fail quietly. A churn rate of 0% or 80% is a data pipeline error, not a real business signal. Catching it in the validation step means you throw an error instead of training a model on garbage data and deploying it to production.
SQL: The Skill That Does Not Go Out of Fashion
Every data scientist works with SQL. The data that models need usually lives in relational databases. SQL is the language of those databases and has been since the 1970s. No amount of Python makes SQL obsolete.
WITH customer_cohorts AS (
SELECT
customer_id,
DATE_TRUNC('month', created_at) AS cohort_month,
created_at
FROM customers
WHERE created_at >= '2022-01-01'
),
monthly_activity AS (
SELECT
c.customer_id,
c.cohort_month,
DATE_TRUNC('month', e.created_at) AS activity_month,
COUNT(*) AS event_count
FROM customer_cohorts c
JOIN events e ON e.customer_id = c.customer_id
WHERE e.event_type = 'login'
GROUP BY c.customer_id, c.cohort_month, DATE_TRUNC('month', e.created_at)
),
cohort_retention AS (
SELECT
cohort_month,
activity_month,
COUNT(DISTINCT customer_id) AS active_customers,
EXTRACT(EPOCH FROM (activity_month - cohort_month)) / 2592000 AS months_since_cohort
FROM monthly_activity
GROUP BY cohort_month, activity_month
),
cohort_sizes AS (
SELECT
cohort_month,
COUNT(*) AS cohort_size
FROM customer_cohorts
GROUP BY cohort_month
)
SELECT
cr.cohort_month,
cr.months_since_cohort::int AS period,
cr.active_customers,
cs.cohort_size,
ROUND(100.0 * cr.active_customers / cs.cohort_size, 1) AS retention_rate
FROM cohort_retention cr
JOIN cohort_sizes cs ON cs.cohort_month = cr.cohort_month
WHERE cr.months_since_cohort >= 0
ORDER BY cr.cohort_month, cr.months_since_cohort;
This cohort retention query answers: "Of customers who signed up in each month, what percentage are still active one month, two months, three months later?" This is one of the most important metrics in subscription businesses. A company with strong month-1 retention but steep decline in months 2-3 has a different problem than one with gradual but consistent decline.
Understanding SQL well enough to write queries like this, and to read execution plans to debug slow queries, is a baseline skill for data science. You will spend hours in SQL for every hour you spend in Python for most roles.
The Tools Beyond Python
Data science work involves more than Python scripts and SQL. The production tooling has matured significantly.
Jupyter is the notebook environment where data scientists do exploratory work. A notebook mixes code cells, output cells, and markdown cells in a single document. You can run cells interactively, visualize data inline, and share notebooks with colleagues who can re-run your analysis. For exploration and communication, notebooks are excellent. For production pipelines, they have serious problems (execution order ambiguity, difficulty testing, poor version control). Use notebooks for exploration. Use Python scripts for production.
dbt (data build tool) is how data engineers build data transformations on top of a SQL warehouse. You write SQL SELECT statements, define dependencies between models, and dbt handles running them in the right order, testing the outputs, and maintaining documentation. For anyone building data pipelines on top of PostgreSQL, MySQL, Snowflake, BigQuery, or similar: dbt is the standard tool.
Apache Airflow schedules and orchestrates data pipelines. You define pipelines as Directed Acyclic Graphs (DAGs) of tasks in Python. Airflow handles scheduling, retries, dependency resolution, and provides a web UI showing pipeline status. For running daily ETL jobs, weekly model retraining, and periodic data quality checks: Airflow is the standard.
MLflow tracks experiments, stores model artifacts, and manages model deployments. When you train 50 variations of a model with different hyperparameters, MLflow logs the parameters, metrics, and the model itself. You can compare runs, register the best model, and deploy it with a consistent interface.
Feature stores like Feast, Tecton, or Databricks Feature Store solve the training-serving skew problem: the features computed for model training need to match exactly the features computed at inference time. Without a feature store, it is common for the offline training pipeline and online serving pipeline to diverge over time, causing model performance to degrade silently.
What Data Science Is Not
Data science is not magic. Models do not discover truth. They find statistical patterns in historical data and assume those patterns will continue into the future. When the patterns change (a global pandemic, a regulatory shift, a competitor entering the market), models trained on pre-change data produce wrong outputs.
Data science is not a substitute for business judgment. A model that recommends price increases based on historical demand elasticity is right in aggregate but wrong for the specific customer who is already upset about last month's price increase. The model does not know what the account manager knows.
Data science is not infallible because the math is rigorous. The math being rigorous is necessary but not sufficient. The data being representative is required. The target variable being a good proxy for the business outcome is required. The features not leaking information from the future into the model is required. The model being deployed to the population it was trained on is required. Any of these being violated produces models that are mathematically elegant and practically useless.
The skill that separates effective data scientists from ineffective ones is not knowing more algorithms. It is knowing where the assumptions break down, what the data quality problems are, and when the model's outputs should not be trusted. That is domain knowledge and statistical judgment combined. No library ships it.
A/B Testing and Statistical Experimentation
Before data science can influence product decisions, companies need a framework for testing whether changes actually improve outcomes. A/B testing is the core methodology. You randomly split users into control and treatment groups, expose treatment to a change, measure outcomes, and apply statistical tests to determine whether observed differences exceed what chance would produce.
from scipy import stats
import numpy as np
import pandas as pd
from dataclasses import dataclass
from typing import Optional
@dataclass
class ExperimentResult:
metric: str
control_mean: float
treatment_mean: float
relative_change: float
p_value: float
confidence_interval: tuple[float, float]
sample_size_control: int
sample_size_treatment: int
is_significant: bool
power: float
def __str__(self):
sig = "SIGNIFICANT" if self.is_significant else "not significant"
return (
f"{self.metric}: {self.control_mean:.4f} -> {self.treatment_mean:.4f} "
f"({self.relative_change:+.1%}) | p={self.p_value:.4f} | {sig}"
)
def run_experiment(
control_data: np.ndarray,
treatment_data: np.ndarray,
metric_name: str,
alpha: float = 0.05,
alternative: str = 'two-sided'
) -> ExperimentResult:
t_stat, p_value = stats.ttest_ind(treatment_data, control_data, alternative=alternative)
control_mean = control_data.mean()
treatment_mean = treatment_data.mean()
relative_change = (treatment_mean - control_mean) / control_mean
se = np.sqrt(treatment_data.var(ddof=1) / len(treatment_data) +
control_data.var(ddof=1) / len(control_data))
z_critical = stats.norm.ppf(1 - alpha/2)
diff = treatment_mean - control_mean
ci = (diff - z_critical * se, diff + z_critical * se)
effect_size = diff / np.sqrt(
(control_data.var(ddof=1) + treatment_data.var(ddof=1)) / 2
)
power = stats.norm.cdf(abs(effect_size) * np.sqrt(len(control_data)/2) - z_critical)
return ExperimentResult(
metric=metric_name,
control_mean=control_mean,
treatment_mean=treatment_mean,
relative_change=relative_change,
p_value=p_value,
confidence_interval=ci,
sample_size_control=len(control_data),
sample_size_treatment=len(treatment_data),
is_significant=p_value < alpha,
power=float(power)
)
def calculate_required_sample_size(
baseline_rate: float,
minimum_detectable_effect: float,
alpha: float = 0.05,
power: float = 0.80
) -> int:
treatment_rate = baseline_rate * (1 + minimum_detectable_effect)
pooled_rate = (baseline_rate + treatment_rate) / 2
z_alpha = stats.norm.ppf(1 - alpha / 2)
z_beta = stats.norm.ppf(power)
numerator = (z_alpha * np.sqrt(2 * pooled_rate * (1 - pooled_rate)) +
z_beta * np.sqrt(baseline_rate * (1 - baseline_rate) +
treatment_rate * (1 - treatment_rate))) ** 2
denominator = (treatment_rate - baseline_rate) ** 2
return int(np.ceil(numerator / denominator))
np.random.seed(42)
n_control = 5000
n_treatment = 5000
required_n = calculate_required_sample_size(
baseline_rate=0.05,
minimum_detectable_effect=0.15,
alpha=0.05,
power=0.80
)
print(f"Required sample size per group: {required_n:,}")
control_conversions = np.random.binomial(1, 0.05, n_control).astype(float)
treatment_conversions = np.random.binomial(1, 0.0575, n_treatment).astype(float)
control_revenue = np.random.exponential(45, n_control) * control_conversions
treatment_revenue = np.random.exponential(48, n_treatment) * treatment_conversions
metrics = [
('conversion_rate', control_conversions, treatment_conversions),
('revenue_per_user', control_revenue, treatment_revenue),
]
results = []
for metric_name, control, treatment in metrics:
result = run_experiment(control, treatment, metric_name)
results.append(result)
print(result)
Sample size calculation before running an experiment is as important as the analysis afterward. Running an experiment without enough data and stopping early when you see a positive result is p-hacking. You will find statistically significant effects in noise. The required sample size formula tells you how long you need to run the experiment to detect an effect of a given size at the desired power level. Run it for that long, then analyze. Do not peek at results daily and stop early.
The effect size matters as much as statistical significance. A 0.01% improvement in conversion rate that is statistically significant is still a 0.01% improvement. Whether it is worth the engineering cost to ship that change is a business question, not a statistics question.
Model Deployment and MLOps
Building a model is 20% of the work in most production ML projects. Getting it deployed, monitored, and maintained is 80%. Data scientists who understand the deployment lifecycle are significantly more effective than those who treat Jupyter notebooks as the end product.
import pickle
import json
import hashlib
from pathlib import Path
from datetime import datetime
import numpy as np
import pandas as pd
from sklearn.pipeline import Pipeline
class ModelRegistry:
def __init__(self, registry_path: str):
self.registry_path = Path(registry_path)
self.registry_path.mkdir(parents=True, exist_ok=True)
self.metadata_path = self.registry_path / 'registry.json'
self.metadata = self._load_metadata()
def _load_metadata(self) -> dict:
if self.metadata_path.exists():
with open(self.metadata_path) as f:
return json.load(f)
return {'models': {}}
def _save_metadata(self):
with open(self.metadata_path, 'w') as f:
json.dump(self.metadata, f, indent=2, default=str)
def register(self, model: Pipeline, model_name: str, metrics: dict,
feature_columns: list[str], description: str = '') -> str:
model_bytes = pickle.dumps(model)
model_hash = hashlib.sha256(model_bytes).hexdigest()[:12]
version = f"v{len(self.metadata['models'].get(model_name, {})) + 1}_{model_hash}"
model_path = self.registry_path / model_name / version
model_path.mkdir(parents=True, exist_ok=True)
with open(model_path / 'model.pkl', 'wb') as f:
f.write(model_bytes)
metadata = {
'version': version,
'model_name': model_name,
'registered_at': datetime.now().isoformat(),
'metrics': metrics,
'feature_columns': feature_columns,
'description': description,
'model_hash': model_hash,
'status': 'staging'
}
with open(model_path / 'metadata.json', 'w') as f:
json.dump(metadata, f, indent=2, default=str)
if model_name not in self.metadata['models']:
self.metadata['models'][model_name] = {}
self.metadata['models'][model_name][version] = str(model_path)
self._save_metadata()
print(f"Registered {model_name}/{version}")
print(f"Metrics: {metrics}")
return version
def load(self, model_name: str, version: str = 'latest') -> tuple[Pipeline, dict]:
if model_name not in self.metadata['models']:
raise ValueError(f"Model '{model_name}' not found")
versions = self.metadata['models'][model_name]
if version == 'latest':
version = max(versions.keys())
elif version == 'production':
production_versions = [
v for v, path in versions.items()
if json.load(open(Path(path) / 'metadata.json'))['status'] == 'production'
]
if not production_versions:
raise ValueError(f"No production model for '{model_name}'")
version = max(production_versions)
if version not in versions:
raise ValueError(f"Version '{version}' not found for model '{model_name}'")
model_path = Path(versions[version])
with open(model_path / 'model.pkl', 'rb') as f:
model = pickle.load(f)
with open(model_path / 'metadata.json') as f:
metadata = json.load(f)
return model, metadata
def promote_to_production(self, model_name: str, version: str):
versions = self.metadata['models'].get(model_name, {})
for v, path in versions.items():
meta_path = Path(path) / 'metadata.json'
with open(meta_path) as f:
meta = json.load(f)
if meta['status'] == 'production':
meta['status'] = 'archived'
with open(meta_path, 'w') as f:
json.dump(meta, f, indent=2)
model_path = Path(versions[version])
meta_path = model_path / 'metadata.json'
with open(meta_path) as f:
meta = json.load(f)
meta['status'] = 'production'
with open(meta_path, 'w') as f:
json.dump(meta, f, indent=2)
print(f"Promoted {model_name}/{version} to production")
class ModelMonitor:
def __init__(self, reference_data: pd.DataFrame, feature_columns: list[str]):
self.reference_stats = self._compute_stats(reference_data[feature_columns])
self.feature_columns = feature_columns
self.drift_alerts = []
def _compute_stats(self, df: pd.DataFrame) -> dict:
stats = {}
for col in df.select_dtypes(include=[np.number]).columns:
stats[col] = {
'mean': df[col].mean(),
'std': df[col].std(),
'p25': df[col].quantile(0.25),
'p75': df[col].quantile(0.75),
'missing_rate': df[col].isnull().mean()
}
return stats
def check_drift(self, current_data: pd.DataFrame, threshold: float = 0.15) -> list[dict]:
alerts = []
current_stats = self._compute_stats(current_data[self.feature_columns])
for feature in self.feature_columns:
if feature not in self.reference_stats or feature not in current_stats:
continue
ref = self.reference_stats[feature]
cur = current_stats[feature]
if ref['std'] > 0:
mean_drift = abs(cur['mean'] - ref['mean']) / ref['std']
if mean_drift > threshold * 5:
alerts.append({
'feature': feature,
'type': 'mean_drift',
'severity': 'high' if mean_drift > threshold * 10 else 'medium',
'reference_mean': ref['mean'],
'current_mean': cur['mean'],
'drift_magnitude': mean_drift
})
if cur['missing_rate'] > ref['missing_rate'] + threshold:
alerts.append({
'feature': feature,
'type': 'missing_rate_increase',
'severity': 'high',
'reference_missing': ref['missing_rate'],
'current_missing': cur['missing_rate']
})
return alerts
A model registry with versioning and promotion workflows is what separates "data science project" from "production ML system." Every model version is immutable and logged. You promote the best version to production explicitly. You can roll back by promoting the previous production version. The monitor detects when incoming data starts looking different from the data the model was trained on, which is the main reason models degrade silently in production.
Should You Go Into Data Science?
If you are comfortable with Python, have a foundation in statistics, and are patient with ambiguous problem definitions, data science is worth pursuing seriously. The demand is real. The compensation is strong. The work is intellectually engaging.
The path in is typically: learn Python well, learn SQL well, work through a statistics course that covers probability, hypothesis testing, and regression, then work on real projects with real messy data. Kaggle competitions give you practice with machine learning but overemphasize leaderboard optimization at the expense of the data engineering, business framing, and communication skills that matter more in real roles.
The tools will change. The fundamentals will not. Statistics, linear algebra, programming, and domain expertise: learn these and the specific tools you need for any given job are learnable in weeks.