Can you use AI with sensitive data? Small models and on-premise AI
Yes. Use a cloud enterprise plan for everyday work and a small model on your own servers for the most sensitive data. Two prompts to copy, then what small models do well and how hybrid works.
Key takeaways
- Start by asking AI: give ChatGPT, Claude or Gemini just the names of the data you use (no contents), have it sort them into Confidential, Internal and Public, and begin with public and de-identified data.
- Small language models (SLMs) run on your own servers, or even a laptop, and do well at well-defined tasks such as classification, field extraction, summarization and rewriting to a template.
- The cost structures differ: cloud AI is pay-as-you-go, while on-premise means a one-time hardware investment plus the people to run it. Test with your own examples before deciding.
- For most businesses a hybrid setup works best: sensitive data is processed on-premise, everyday tasks go to large cloud models, and an integration layer routes and logs every job.
Table of contents
Yes. Everyday work can go through a cloud AI enterprise plan, since the major providers usually commit to not training on your data and to encrypting it in transit. The data that must not leave can go to a small language model (SLM) on your company's own servers.
"Our data is sensitive. It can't leave the company." That's often the first thing businesses say when they start looking at AI, and it's a fair point: quotes, formulas, customer lists, design drawings, medical records and financial data should be handled with care. Below are two prompts to start with that need nothing confidential.
Ask AI first
Paste them into an AI assistant such as ChatGPT, Claude or Gemini, swapping in your own details (the ones below are examples).
Sort your data into three levels
We're a metal parts factory. Below are the types of data we use (names only, no contents): quotes, cost sheets, customer lists, design drawings, process parameters, payroll, product catalogs, website copy. Sort each one into Confidential, Internal or Public, with one line on why (for example, personal data law, a customer NDA or company policy), and suggest which can go to cloud AI, which should be de-identified first and which should stay in-house. Give me a table I can paste into Excel.
Build a test set
I want to test whether AI can sort our customer emails into quote requests, complaints, shipping questions and other. Below are 20 de-identified examples (customer names replaced with codes, phone numbers and addresses removed). Classify each one with a one-line reason, then lay them out as a test table: example number, summary, AI's category, correct answer (leave blank, I'll fill it in), correct or not.
The levels are a draft: check your contracts and the rules to see which data is restricted, and have someone who knows the work fill in the correct answers. Until you've confirmed your company is on an enterprise plan, keep the contents of quotes, formulas and customer lists out of cloud AI.
5 things you can do now
1. Sort your data into three levels
Have the person responsible check each line of the table from the first prompt and note the reason: a regulation, a customer contract or company policy.
2. Start with public and de-identified data
Product introductions, published specs and copy already on your website make a good first job for AI to rewrite and tidy up. De-identify internal data before you use it: replace names with "Customer A" and delete phone numbers and addresses. In Excel, Find and Replace can swap every instance of a name at once.
3. Check what your cloud plan covers
Find out whether colleagues use personal accounts or an enterprise plan, then ask the provider four things: is our data used for training, how long is it kept, which region is it stored in, and can we ask for it to be deleted?
4. Build your own test set
Pick one task you'd like AI to handle and collect 50–100 real examples with the correct answers (de-identify any you'll test in the cloud, and lay them out with the second prompt). Whenever you test a cloud or on-premise model, use this set to compare accuracy (and where it goes wrong), speed (on the hardware you plan to use, with several people at once), consistency (same result and format every time) and language ability (models vary a lot with Traditional Chinese and Taiwanese terms). Online benchmark scores are a useful reference, but your own data tells you the most.
5. Decide who will run it
On-premise AI needs someone to deploy, monitor and update it and deal with failures: a colleague or an outside partner. Until you have someone, you can use a cloud enterprise plan and keep building up test results.
What is a small language model?
The large language models people usually talk about are cloud services from AI companies, with enormous numbers of parameters and broad capabilities. Small language models have far fewer parameters, and what sets them apart is:
- They run in your own environment: company servers, workstations, even laptops.
- They're fast and their costs are predictable: there's no per-call fee.
- They can be tuned for specific tasks: fine-tune a model on company data so it learns the vocabulary of your field.
Common open-weight model families include Llama, Qwen, Gemma and Mistral, each in a range of sizes. Every model has its own license terms, so check them before any commercial use.
How do large cloud models and small on-premise models compare?
| Comparison | Large cloud models | Small on-premise models |
|---|---|---|
| Where data goes | Sent to the provider for processing (enterprise plans usually include no-training, encryption and similar safeguards) | Stays entirely inside the company |
| Capability | Broad; strong at complex reasoning, long documents and multi-step tasks | Good at well-defined tasks; weaker at complex reasoning |
| Cost structure | Pay-as-you-go, low upfront cost | One-time hardware investment, plus power and upkeep |
| Operations | Handled by the provider | Someone has to deploy, monitor and update it |
| New models | Available as soon as they launch | You evaluate, test and swap them in yourself |
What are small models good for?
Small models do well on well-defined tasks that have a clear right answer:
- Classification and routing: is this email a quote request, a complaint or spam?
- Field extraction: pulling customer names, part numbers, quantities and dates out of documents (see reading orders and quotes with AI).
- Summarization: condensing meeting notes or customer service chats into the key points.
- Rewriting to a template: turning technical specs into descriptions customers can follow.
- Pre-processing sensitive data: masking personal information on-premise first, then passing the cleaned-up content to a large cloud model.
Tasks that need broad knowledge, multi-step planning or long stretches of autonomous work, such as an AI agent handling a complex process across several systems, are still more reliable with large cloud models.
How do you decide which to use?
Use your preparation to answer three questions:
- Do regulations, customer contracts or company policy keep this data inside the company? (Step 1.) If so, process that part on-premise.
- Is the task well-defined, with a clear right answer? (Step 4.) If so, a small model is usually enough; for complex reasoning, consider a large cloud model or a hybrid setup.
- Do you have someone who can run it? (Step 5.) If not yet, work with a partner, or start in the cloud.
When to consider the next step
With these steps in place, a lot of work can be handled with confidence on a cloud enterprise plan. Consider bringing part of your AI in-house when:
- More of the work you'd like AI to do involves data marked Confidential.
- You want to analyze customer service logs or other personal data, and de-identifying it takes longer and longer.
- Your test set shows that a small model is good enough for your task.
The next step: a hybrid setup that keeps sensitive data in-house
In practice, most businesses are best served by a hybrid architecture: sensitive data is processed on-premise, everyday tasks (writing copy, analyzing public or de-identified data) go to large cloud models, and an integration layer sends each job to the right model based on the data and task, logging every run. You don't need a big hardware investment on day one.
Take analyzing a year of customer service conversations, which contain names, phone numbers, addresses and order details (illustrative):
- A small on-premise model goes first: on a server inside the company, it masks personal information (names become "Customer A," phone numbers and addresses are removed) and sorts the conversations into shipping, quality, refunds and other.
- Only de-identified content goes to the cloud: the masked summaries go to a large cloud model, which finds root causes and suggests improvements.
- The results come back inside the company: the report is stored in your own systems, with a record of which model processed which data each time.
The key design decision is working out what information may leave the company before any data does.
When you roll it out, compare cloud and on-premise models on your test set from step 4, then decide how to split the work. Count power, server room space, staff time and updates in the cost (see the reasons for local AI and how to run the numbers). During the pilot, keep working as you do now, have a person check the AI's results, and keep models easy to swap.
Data that can't leave the company doesn't have to mean no AI. To find out which approach fits your data and tasks, start with a free AI Health Check, or take a look at our AI integration services and custom business systems. You're also welcome to tell us what you need.
