If your line of business involves corporate investing, wealth management, real estate, or raising capital, keeping your analytics on the leading edge is crucial. While in 2024, “leading-edge” typically means cloud, there’s a lot of room for innovation for solutions that need to run on-premises due to industry constraints.
One of our latest FinTech projects is just like that. In an engagement with a leading wealth management SaaS, we developed an analytics engine that classifies 7+ million financial instruments within a two-hour window. Leveraging a microservices-based architecture with MongoDB and Java Spring as core technologies, we’ve made the solution extremely adaptable to the increasingly fluid financial assets market.
This article will outline the reasoning behind our experts’ principal architecture decisions taken to address project goals and constraints. If building analytics software is among your professional interests, read on for a quick overview of how we do it at AgileEngine.
Project overview: ensuring precision at scale
The firm that engaged our engineering solutions team provides wealth management recommendations to thousands of customers worldwide. The range of financial instruments covered by its platform encompasses the entire U.S. stock and bond markets, constituting seven to eight million securities on an average day.
“How does the firm decide which securities to recommend for a given portfolio?” you might ask. A significant part of the recommendation logic relies on the precise classification of tradable financial assets known as securities. In a nutshell, the firm assigns each security to an asset class — like, for instance, “domestic small-cap corporate stock” — and forecasts the future performance of a given class or asset.
Classifications of this kind typically leverage custom logical rules written by the firm’s internal financial experts. Due to market volatility, classifications need daily updates to remain valid and relevant. Also, due to our client’s operational constraints, the entire set of 7–8 million assets had to be classified within a two-hour window. The project also included additional industry-specific and company-specific constraints impacting our choice of technology and architecture.
Like this post? You’ll like our deliverables even more! Engage top-1% software development, UI/UX, data, and AI professionals.
Accounting for our client’s technical constraints
As a financial services firm, our client needed to comply with strict regulations limiting the range of technologies available for building its software systems:
- On-premise solutions: The company’s policy required all applications to run on-premises, which excluded public clouds like Amazon Web Services.
- Open-source code: Our client’s policy also limited the choice of technologies to open-source code frameworks and libraries pre-approved by the company’s application security committee.
- Budget: The client also had a limited budget for hosted infrastructure like web servers and databases.
So how do you analyze 7+ million assets in under two hours, using an on-premise infra built on a budget? In a single word, the answer is efficiency. In order to meet our client’s requirements and constraints, our implementation had to conserve on-premise instances while utilizing available resources.
Our recommendations: microservices, Java Spring, and MongoDB
After assessing the client’s business requirements and technical constraints, our solution architects wrote a comprehensive technical proposal covering the classification engine’s core technologies, system architecture, and data flow.
Architecture: microservices with messaging broker
Stability and availability are critical for the analysis engine, so we avoided bundling it together with related functionality, such as the API, in a monolithic architecture. Instead, our architecture decomposed the solution into separate microservices for each major pipeline stage. This approach helped us keep the engine online even if bugs or crashes occurred in other system parts. In our solution, each microservice used a messaging broker to trigger other microservices.
Programming language and framework: Java Spring
Java and the Spring framework have a solid track record in enterprise development under strict security requirements and a well-established, highly active community. Many of our client’s developers have extensive experience with these tools. Spring was also optimal for quick and efficient implementation of various application functionalities, such as database interactions and HTTP security.
Database: MongoDB
The properties defining a typical financial instrument change over time. As a result, our project required a flexible data model where properties could be added and removed. We recommended MongoDB as an industry-leading open-source NoSQL database capable of storing data with an evolving structure.
Messaging broker: RabbitMQ
We chose RabbitMQ as a performant and reliable open-source library for passing messages between the microservices. RabbitMQ’s ability to deploy multiple instances on-premises made it highly cost-effective. Thanks to the replication of messages across clusters, RabbitMQ also helped us ensure that pipeline stages reliably followed a correct sequence, even in the event of temporary application failures.
CI/CD pipeline: Bitbucket and Bamboo
Our client’s use of Atlassian Jira for project management and task tracking made choosing CI/CD tools easy. Being part of the Atlassian suite, Bitbucket and Bamboo became our go-to options for code management, build automation, and deployment control, complete with status tracking via a Jira integration. The maturity of these three platforms made for an excellent package with all the functionality we needed.
Monitoring: Splunk
A high-throughput data processing pipeline wouldn’t be complete without granular near-real-time monitoring. Any known defect in the output stream and any exception in the execution should be logged, monitored, and triaged as necessary. Such pipelines can also produce a massive volume of logs, making data visualization essential. Splunk is a robust tool that provides all of these capabilities, so we rounded out our proposal by recommending it and suggesting how a visual dashboard would be effective.
Solution delivery: it’s all about people
The classification solution was part of a broader engagement where our experts collaborated with the client’s developers as a remote dedicated team. With this in mind, our technology decisions also needed to consider whether the client’s in-house team would be comfortable developing and maintaining the solution. At the end of the day, technology is about people, and technology decisions need to reflect this.
Working alongside the client’s business and technical stakeholders, our team refined the technical design with greater fidelity, configured the environments and tooling, and wrote most of the application code. The resulting classification system proved to be one of the most successful solutions delivered by our experts within our two-year engagement with the company.
Summing it all up: the art of making the right decisions
While not as easy to measure as code quality or development speed, decision-making is just as integral for product success. By making the right decisions in areas like architecture, technology, and process, you can prevent costly errors and rework while matching project parameters to business needs.
At AgileEngine, new projects are often taken up by starter leads — our top architects with over a decade of experience in their fields. When recommending architectures and technologies to clients, our starter leads will typically consider factors like industry and business context, as well as internal team composition, existing technology, and more.
Following this approach has consistently enabled our teams to build products that get top spots on rankings like Gartner, G2, and Capterra. If you’re interested in exploring how a similar strategy can set your digital products up for success, feel free to contact us and share your challenge with our experts.
