AI Lead Scoring: Building a Model That Learns

AI lead scoring is a model trained on your own conversion history that outputs, for each lead, the probability that this lead converts. It replaces the point table, fifteen points for a pricing page visit, ten for a whitepaper, twenty for a VP title, with weights the data chose. Five pieces make one up: features engineered from behavior, a target variable that names what counts as a conversion, a model, a threshold that turns a probability into a routing decision, and a delivery path that puts the number where reps work. The difference that matters is not the algorithm. It is that nobody is guessing what a pricing page visit is worth.

Point-based scoring fails quietly. The weights were set once in a room by people with good instincts and no measurement, buyer behavior moved, nobody recalibrated because recalibrating was manual, and eventually the sales team stopped sorting by the score. A model that retrains on recent conversions does not drift the same way. This guide covers the features worth engineering, how to pick a model for your data volume, how to set thresholds, and how to wire scores into the CRM so they change what a rep does on Monday.

Why Point-Based Scoring Fails

To see why a learned model beats a point table, it helps to be precise about where the point table breaks. The failures are structural, not the result of anyone doing it badly.

The Weights Are Guesses

In a rules-based model, every weight comes from human judgment, which is why deciding which metrics actually matter has to happen before any scoring work. A committee decides a pricing page visit is worth fifteen points. But a pricing page visit from someone who already completed activation means something entirely different from a pricing page visit in the first session, where the person is just orienting. A static table assigns the same fifteen points to both.

The Math Is Additive and Buying Is Not

Point tables add. That assumes more behavior always means more likelihood, in a straight line. Real journeys have thresholds: someone who invites three teammates is in a different situation from someone who invited one, while the difference between four and five invites is nothing. They have interactions: pricing after activation predicts conversion, pricing without activation predicts browsing. Addition cannot represent either.

It Decays and Nobody Notices

Behavior shifts. A signal that predicted conversion two years ago may be meaningless now. New product features create signals the old table has no row for. A campaign changes the mix of who enters the funnel. Static models degrade silently, because prediction accuracy is not being measured, so the first visible symptom is reps openly ignoring the number.

Features: Turning Behavior Into Model Inputs

Feature engineering turns raw behavior into inputs a model can learn from, and it sets the ceiling on accuracy. Better features on a simple model beat a sophisticated model on poor features, reliably. This is where the analytics layer earns its place: you need granular, person-level history to engineer anything worthwhile.

Engagement Features

Counts like “sessions” and “pages viewed” are a starting point and not much more. Engineered versions predict far better: session frequency trend, whether usage is rising, flat, or falling week over week; time to first meaningful action after signup; depth per session rather than visit count; and recency-weighted engagement, where last week counts more than three months ago. These capture trajectory, which is what actually distinguishes a lead who is warming from one who is fading.

Product Usage Features

For software, usage features tend to be the strongest predictors available. Adoption breadth, how many distinct features the person has tried. Adoption depth, whether they return to what they tried. Workflow completion rate across multi-step processes. Collaboration signals such as invites sent and shared objects created. In Kissmetrics, these live as events and properties on the person profile. Autocapture matters here more than it sounds: features you never thought to instrument still produce signal, which means the model can find predictors you would not have thought to write down.

Intent Features

Intent features are behaviors that point directly at a buying decision: pricing page frequency and recency, comparison page visits, case study and ROI calculator consumption, admin and billing page views, plan selector interactions. Intent features usually carry the most predictive power per event, and they are also the rarest. That combination is why they cannot carry a model alone. Pair them with engagement and usage features so the model has something to read on the many leads who never touch a pricing page.

Firmographic and Contextual Features

Behavior should dominate, but company size, industry, role, region, and acquisition source add real context. The point is to let the model decide how much they are worth rather than hard-coding the answer. A model may find that company size predicts strongly in one industry and not at all in another, which is exactly the kind of conditional a point table cannot express.

Choosing a Model

Model choice follows from data volume, engineering capacity, and how much you need to explain a score to a skeptical sales director. There is no best algorithm here, only a trade between accuracy, interpretability, and how much of your life the thing consumes.

Logistic Regression

The simplest approach that works, and the right starting point for almost everyone. It models conversion probability as a function of your features and returns a number between 0 and 1 you can read as a probability. It trains in seconds, it survives small datasets, and you can explain any individual score by pointing at the coefficients. If you have a few hundred conversions in your history, this will already tell you things your point table never could.

Gradient Boosted Trees

XGBoost and LightGBM are the default for tabular problems. They pick up non-linearity, interactions, and threshold effects without you specifying any of them, which addresses precisely the failures that sink point tables. The cost is explainability: answering “why did this lead get a 0.81” takes SHAP values rather than a glance at a coefficient. Worth it once you have a few thousand conversions and someone who can maintain a training pipeline.

Neural Networks

Deep models can represent sequence, the order of actions rather than their counts, which genuinely matters in some funnels. They also want tens of thousands of conversions and a real ML engineering practice to train, deploy, and monitor. For most B2B lead scoring this is the wrong trade. Consider it only with large datasets, strong sequential structure, and people whose job this is.

Training on Your Conversion Data

Training needs historical leads with known outcomes, converted and not converted. A few decisions at this stage matter more than the algorithm you pick after them.

Define the Target Variable

What counts as a conversion depends on the business. For self-serve software the obvious target is a paid subscription, but free to trial, trial to paid, and paid to expanded are all valid and produce different models. A free-to-trial model is a marketing prioritization tool. A trial-to-paid model is a sales prioritization tool. Training separate models per stage is usually better than one model asked to do both jobs.

Set the Observation Window

The observation window is the period over which you measure behavior before the prediction. All-time data drags in behavior from long before the person was considering anything. Seven days misses early signal. A 30-day rolling window is a reasonable starting point for software, but test 7, 14, 30, and 60 and let the validation numbers decide. This requires timestamped person-level events you can aggregate over any window, which is what Kissmetrics metrics are built on. If you need a specific windowed aggregate, ask the AI chat for it in a sentence. It builds the metric against events Kissmetrics captured itself, works out the windowing internally, and saves the definition so the training job can re-run the same thing next quarter.

Handle Class Imbalance

Conversions are rare. At a 3% conversion rate your training data is 97:3, and a model trained on it without adjustment learns to predict “no” for everyone, which scores 97% accurate and is completely useless. Fix it by oversampling the minority class, undersampling the majority, adjusting class weights, or evaluating on precision-recall AUC instead of accuracy. Most gradient boosting libraries support class weights directly, and that is usually the least disruptive option.

Validate on Time, Not at Random

Never evaluate on the data you trained on, and do not split randomly. Split by time: train on months 1 through 9, validate on months 10 through 12. That mirrors deployment, where the model scores future leads using past patterns. Random splits leak temporal information and flatter the model.

Thresholds and CRM Integration

A model living in a notebook is worth nothing. Value appears when the score reaches the place decisions get made, which for most teams means the CRM, with thresholds attached that change what happens to a lead.

Setting Thresholds

A probability is not a decision. The threshold is what converts 0.62 into “a rep calls this person today.” Set it from sales capacity rather than from a round number: if your team can work 200 leads a week, the top band should contain roughly 200 leads a week. Then check the band, not the model, by asking what share of the leads above the line convert compared with the overall rate. If that gap is not clearly worth a rep’s time, the threshold is in the wrong place or the model is not ready.

Two or three bands is usually enough. Route the top band to a rep with a task attached. Put the middle band into nurture and let it re-score. Leave the bottom band alone, and be explicit that this is the point, since the value of scoring comes as much from what reps stop doing as from what they start.

Delivering the Score

The score belongs on the lead or contact record as a normal field, sortable and filterable, with saved views that surface the top band. Refresh daily at minimum. Behavior changes fast, and a three-day-old score can be describing someone who has since gone quiet or someone who spent the weekend in your product.

Then attach triggers. Crossing into the top band assigns an owner and creates a task. Dropping below the line moves the lead to nurture. A sharp jump in score, a burst of activity rather than a gradual climb, alerts the owner while the interest is live. Without triggers the score is a column nobody sorts by.

Keeping the Model Current

A scoring model is not a project that finishes. What predicts conversion moves as the product changes, the market changes, and the customer base changes. An unmaintained model decays the same way a point table does, only more slowly and with more dignity.

Monitor Lift, Not Accuracy

Accuracy is misleading on imbalanced data. The practical metric is lift: the conversion rate of the top decile of scores against the overall conversion rate. The absolute number depends on your business, so the thing to watch is your own trend. Chart it monthly. When lift falls, the model needs retraining, and you will know before the sales team tells you.

Retraining Cadence

Quarterly retraining suits most B2B teams: often enough to track real change, rare enough to stay operable. Retrain on the most recent 12 months, or whatever window holds a representative set of conversions. Compare the candidate against the incumbent on the same validation split and only ship it if it wins. Swapping models that score the same creates churn in every downstream workflow for nothing.

Refresh the Features Too

Retraining should include a feature pass. Are there new product features generating signal the model has never seen? Are there features that no longer mean anything because the product changed underneath them? Add, remove, and re-rank by importance. The largest improvements often come from one new feature covering a dimension of behavior nothing else measured.

Human Oversight and Calibration

Models learn whatever is in the history, including the parts you would not have chosen. They overfit to patterns that have expired. They produce scores that are technically calibrated and practically misleading. Oversight is how you catch that.

Review the Surprises

Run a monthly review with sales and marketing. Pull high-scoring leads that did not convert and low-scoring leads that did, and work out why the model was wrong. Those cases are where the missing feature or the learned bias shows up. If the model keeps overscoring one acquisition source because that source used to convert well and recently stopped, a human catches it in a month rather than a quarter.

Let Reps Override

A rep who has spoken to a lead knows about timeline, budget, and authority, none of which appears in behavioral data. Give them an override and store both numbers, the model score and the adjusted one. The gap between them is free training signal: patterns in what reps consistently correct tell you what feature to add next.

Audit for Bias

If your team historically closed more deals in certain industries or company sizes because that is where they spent their effort, not because those leads were better, the model learns the effort pattern and reinforces it. Check calibration across company size, industry, and geography periodically. Where predictions are systematically off for a group, correct it in the training data or with model constraints rather than by explaining it away.

What a Working Model Looks Like

The honest version of the results section is a description of what changes, not a percentage from someone else’s deck. Here is what to expect if the model is working, and how you would see it in your own numbers.

The Top Band Feels Different to Reps

When weights come from conversion data rather than a committee, the top band fills with people who have actually used the product rather than people who match a profile. Reps notice this before any chart does, because the conversations start further along. The measurable version is top-decile lift against your own baseline, tracked month over month.

Leads That Do Not Fit the Profile Start Surfacing

Point tables encode who you think buys, so they can only ever find more of that. A model that reads behavior surfaces the small company with intense product usage that the firmographic filter was discarding, and deprioritizes the enterprise logo that has logged in twice. The mechanism is simply that behavioral features outweigh profile features when the data says they should.

Expansion Signals Arrive Earlier

In accounts you already have, usage shifts often precede an expansion conversation by weeks. A model reading product usage detects the change on the week it happens rather than at the next quarterly review, which gives the account manager the chance to open the conversation instead of responding to it. Quantify it with customer lifetime value, since the value of an earlier expansion is the revenue pulled forward plus the deals that would not have happened at all. For more on reading behavior forward in time, see behavioral data predictions.

How to Prove It

Run the old scoring and the new model side by side for a quarter without telling reps which is which, then compare conversion rate within the top band of each. That is the only comparison that controls for the fact that better-prioritized reps also work harder. Skip it and you will never know whether the model helped or the team just had a good quarter.