What will software engineering actually look like when enterprises begin applying AI to their own private data? I have been trying to answer this question and this post I am laying down what I have understood so far.

The private data enterprises already own data powers their processes, systems, decisions, and customer interactions—and it is the natural foundation for AI that could create value or reduces operational cost.

Two Categories of Enterprise Private Data

Enterprise data broadly falls into two buckets, each leading to a different type of AI application deployment.

1. Data Managed by Third-Party Enterprise Products

Most enterprises run a large portion of their business on third-party SaaS platforms—Salesforce, Workday, SAP, ServiceNow, Jira, Box, Slack, and others. With these systems, two scenarios emerge.

1A. Vendors Provide AI Features Directly

Many product vendors are now integrating their own AI capabilities into their suites. Examples include:

• Salesforce Einstein
• ServiceNow Now Assist
• Atlassian Intelligence
• Google Workspace Duet AI

In these cases, enterprises primarily configure, adopt, and govern the AI features, but do not build them.

1B. Vendors Provide APIs / MCP Interfaces, but No End Use AI Layer

Some products won’t ship AI capabilities—and will only expose APIs, events, and MCP servers over the data they store.

Examples:

• Google Drive / Box exposing document APIs
• Slack providing conversation history APIs
• Ticketing systems exposing workflow events
• Internal productivity tools providing MCP-compatible interfaces

Here, enterprises will build their own AI applications on top of these data sources, using:

• RAG (Retrieval-Augmented Generation)
• Tool calling/MCP
• Agent workflows
• Automation pipelines

2. Data Belonging to Custom Internal Systems

Every enterprise has significant proprietary systems:

• Operational databases
• Internal microservices
• Event logs
• Document archives
• Domain-specific workflow systems
• Analytics warehouses

These are prime candidates for custom AI development, either to reduce cost (automation) or to create new capabilities for employees and customers.

What Enterprise AI New Software Development May Look Like

As enterprises begin to apply AI to their private data, certain engineering patterns will become standard. Below is a picture of what development teams may actually do.

1. Data Ingestion and Vector Indexing

Data pipelines that:

• Ingest documents, tables, logs, chats, emails, and artifacts
• Chunk and embed unstructured data
• Maintain vector databases or hybrid (lexical + semantic) indexes
• Continuously update embeddings as data changes

Note: RAG does not always require a vector database—lexical search, APIs over classical search engines may be applicable.

2. Unstructured Data / Document Processing

A lot of enterprise data is unstructured, so teams will develop workflows for:

• Document parsing (PDF, email, HTML, scans)
• Table extraction
• Metadata enrichment
• Summaries, Q&A, and semantic retrieval

Unstructured data processing may become a core competency.

3. AI Tooling and MCP Development

Enterprises may increasingly expose internal systems as AI tools:

• Querying customer records
• Updating tickets
• Running analytics queries
• Triggering workflows
• Interacting with micro-services
• Executing controlled code in sandboxes

These tools form the “API layer” for agentic systems. The emerging MCP standard helps this trend.

4. Agent Assembly and Workflow Engineering

These include:

• Main orchestrator agents
• Sub-agents supporting the main agents
• Tool selection and routing
• Multi-step workflows
• Safety gates and verification layers
• Error recovery, retry logic, and fallbacks
• Optional human-in-the-loop steps

5. Prompt Engineering as System Design

Prompt engineering will be a particularly interesting software engineering discipline. Dealing with non-deterministic nature of AI models feels like humanities meeting programming. Shaping context windows and guiding tool-invocations using prompts is going to be a key activity that determines the quality of output of AI applications. It is the prompts that marry the deterministic world with AI.

6. Integrating Open-Source AI Tools

Enterprises will likely assemble an ecosystem of tools for:

• Web search
• Lexical/semantic search
• Data scraping (internal)
• SQL and NoSQL querying
• Code generation and safe execution
• Mathematical computation
• Document generation
• Connectors for services like Drive, Box, Slack
• Knowledge base extraction (e.g., Wikipedia, PubMed, finance, maps)

This assembly pattern looks much like building with micro-services, or independent unix utilities piped together.

7. Choosing the Right Models

Model strategy is another interesting architectural decision:

• Proprietary foundation models (GPT, Claude, Gemini)
• Open-source foundation models (Llama, Mistral)
• Domain-specific models (code, vision)

This involves engineering around:

• Balancing cost vs. performance
• Privacy and data residency
• On-prem vs. cloud deployment
• Evaluating model correctness
• Choosing right model for the task.

It is quite possible that we have a lot of special models to choose from to put together an AI application.

What I Have Left Out Here

• Quality testing and evaluation
• Security and compliance
• Performance tuning
• Observability
• Cost control
• Data governance

Not because they are unimportant—but because they are cross-cutting concerns that apply to all software engineering - and they will continue to.

I’ve also skipped fine-tuning and model training, mainly because they require different expertise and I don't understand much right now about these.

This post is a prognosis based on my line of sight and understanding so far. It is not meant to be a prediction but rather food for thought.

Lastly along these line I am liking the book - AI Engineering - seems to be dealing with the same topic. Feels like written for people with software engineering background. Another remarkable resource for engineers is the engineering blog of anthropic.