Tomcat
Professional
- Messages
- 2,695
- Reaction score
- 1,072
- Points
- 113
The main thing in the payment system is to take money, transfer records from one plate to the same plate with a minus sign. It doesn't sound very difficult until the lawyers came. Payment systems all over the world are subject to a huge number of all kinds of burdens and instructions. Therefore, as part of the development of a payment system, you have to constantly balance on the verge between a heavy enterprise and a completely normal scalable web application.
Under the cut, the story of Philippe Delgado (dph) on Highload ++ about the experience accumulated over several years of working on a payment system for the Russian legal betting business, about mistakes, but also about some achievements, and about how to competently mix, but not shake up, the web with the enterprise.
About the speaker: Philippe Delgado has done a lot of things during his career - from two links in
Visual Basic to hardcore SQL. In recent years, he has been mainly engaged in loaded projects in Java and regularly shares his experience at various conferences.
We have been making our payment system for three years, of which we have been in production for two years. Two years ago I told you how to make a payment system in one year, but since then, of course, a lot has changed in our solution.
We are a rather small team: 10 programmers, mostly back-end developers and only two people on the front-end, four QA and me, plus some kind of management. Since the team is small, there is not much money, especially in the beginning.
In reality, this is how it is, the payment system is a very simple thing. Until the lawyers came . Payment systems all over the world are subject to a huge number of all kinds of burdens and instructions on how to transfer money from one tablet to another correctly, how to interact with users, what they can be promised, what cannot be promised, what we are responsible for, what we are not responsible for. Therefore, as part of the development of a payment system, you have to constantly balance on the verge between a heavy enterprise and a completely normal scalable web application.
From the enterprise we have the following.
We work with money.
Therefore, we have complex accounting, we need a high level of reliability, a high SLA (because neither we nor the users like a simple system), and high responsibility - we must know exactly where and what user's money went and what happened to them. the moment generally happens.
We are an NPO (non-bank credit institution).
It is practically a bank, but we cannot issue loans.
We have lawyers. Here are some of the laws that govern the behavior of payment systems at the moment in the Russian Federation:
Each of these laws is quite difficult to implement, because there are still a bunch of bylaws, real use stories, and you have to work with all this.
At the same time, besides the fact that we are such a pure enterprise, almost a bank, we are still quite a web company.
We use Java because it is somewhat easier to find developers on the market who at least roughly understand about the reliability of working with databases in the Java world than in other languages.
I know only three databases with which you can make a payment system, but one of them is very expensive, for the second it is difficult to find support and it is also not free. As a result, PostgreSQL is the best option for us: it's easy to find sane support, and in general, for little money, you don't have to think about what is happening with the databases at all: everything is clean, beautiful and guaranteed.
The project uses a little Kotlin - rather for fun and to look to the future. So far, Kotlin is mostly used as some scripting language, plus some small services. Service architecture
, of course . It is categorically impossible to call it microservices. Microservice, in my understanding, is something that is easier to rewrite than to understand and refactor. Therefore, we, of course, do not have a microservice, but normal full-fledged large components.
In addition, Redis for caching, Angular for backend. The main site, visible to the user, is made on pure HTML + CSS with a minimum of JS.
And of course, Kafka.
But security requirements come . We have personal data in the system, personal data must be stored and processed separately, with special restrictions. We have information on bank cards, it must also live in a separate part of the system with the appropriate audit requirements for each code change and requirements for data access. Therefore, you have to cut everything into components.
There are requirements for reliability... I don’t want to, because some gateway with one of the counterparty banks for some reason broke down, take and lay out all the payment logic: God forbid, there will be an incorrect calculation, the human factor - and everything will collapse. Therefore, you still have to divide everything into relatively small services.
But since we are starting to divide the system into components according to security and reliability requirements , it makes sense to separate out into separate services everything that requires its own storage. Those. for a part of the database that does not depend on the whole other system at all, it is easier to create a separate service.
"Processing" or "Reports" is a more or less normal name, "Bullshit that works with a database replica" is already a bad name. Obviously, this is not one service - either several, or part of one large one.
These four requirements are quite enough for us to allocate separate
services.
Of course, we still have micromonoliths that we keep cutting because they accumulate too many words and too many names for one service. This is an ongoing process of redistribution of responsibility.
The services themselves interact via JSON RPC over http (s), as in any web world. At the same time, for each service, a separate logic for retrying requests and caching results is prescribed. As a result, even if a service crashes, the entire system continues to work normally, and the user does not notice anything.
Well, among other things, this is a backup source of data about all our operations.... I am, of course, paranoid, since the specifics of the work are conducive. I saw (quite a long time ago and not in this project) how a commercial database for a lot of money at some point began to write nonsense not only to its own database files, but also to all replicas and backups. And the data had to be restored from the logs, because it was data on the payments made, and without them the company could safely close the next day.
I don't really like pulling out important data from the logs, so I'd rather put all the necessary information into the same Kafku. If suddenly some impossible situation happens to me, I, at least, know where to get the backup data from, which has nothing to do with the main storage.
In general, for payment systems to have two independent data storages is a standard practice, living without it is simply scary - for me, for example.
What is the problem?
Therefore, I seriously think to go somewhere.
Further, I will tell several separate cases about what architectural challenges we had, how we solved them, what was good and what was bad at the same time.
To be honest, there will be many tables, many relationships between them. As a result, you need an ORM, you need complex migration logic when adding a column. Let me remind you that in PostgreSQL, even a simple addition of a new nullable column to a table can lead (in some specific situations) to the fact that for a long time this table will not be available at all. Those. in fact, adding a nullable column is not an atomic free operation, as many people think. We even stumbled upon it once.
All this is rather unpleasant and sad, I want to avoid all this, especially when using ORM. Therefore, we remove all these large and complex entities in JSON, simply because in reality, except on the application server, all these data and structures are not needed anywhere. I have been using this approach for 10 years and, finally, I notice that it is becoming, if not mainstream, then at least a generally accepted practice.
First, you must immediately think about possible conflicts.
Once you had a version of an object with one set of data fields, you released another version, where there is already a different set of data fields, you need to somehow read the old JSON and convert it into an object convenient for you.
To solve this problem, it is usually enough to find a good serializer / deserializer, to which you can explicitly tell that this field from JSON needs to be converted into such and such a set of fields, these things should be serialized this way, and if something is not there, then replace with default, etc. In Java, fortunately, there are no problems with such serializers. My favorite is Jackson.
Be sure to store the version of the structure you are writing in the database.
Those. next to each field where you store your JSON, there should be another field where the version is stored. First of all, this is necessary in order not to endlessly support the code of understanding the old version of the new one.
When you release a new version and you have a new data structure, you just make a migration script that runs through the entire database, finds all the old versions of the structure, reads them, writes them in the new format, and after some rather limited time you have , a maximum of 2-3 different versions of the data remains in the database, and you do not bother with the support of all the diversity that you have accumulated over the years. It is ridding yourself of legacy, ridding yourself of technical debt.
For PostgreSQL, you have to choose between json and jsonb.
Once upon a time, this choice still made sense. For example, we used JSON because we started a long time ago. Let me remind you that the JSON data type is just a text field, and PostgreSQL will parse it every time to get inside. Therefore, in production it is better not to get inside the json objects in the database once again, only in case of some kind of support or troubleshooting. In an amicable way, your SQL code should not contain commands for working with json fields at all.
If you use JSONB, then PostgreSQL neatly parses everything into a binary format, but does not preserve the original form of the JSON object. When we, for example, store the original data coming to us, we always use only JSON.
We don't need JSONB yet, but at the moment, it really makes sense to always use JSONB and not think about it. The difference in performance has become practically zero, even for simple read and write.
The reliability of the service was realized through active-standby - because the service is small, it restarts quickly, other components will wait for 3-5 seconds, so there is no point in piling up some complex cluster system.
Before launching, we began to undergo a PCI DSS audit, and it turned out that there are rather strict requirements for data access control, which, in the case of our auditor, boiled down to the following:
To begin with, we stop trusting the operation manager and try to come up with a scheme when we do not have one person who knows the keys.
Logically, we come to Shamir's scheme. This is a way of generating a key, when several keys are generated based on the ready key, any subset of which can generate the original key.
For example, you form a long key, immediately break it into 5 pieces so that any three of them could generate the original one. Then you distribute these three exploits, store two in a safe, just in case someone gets sick, gets hit by a bus, etc., and live peacefully. You no longer need the original long key, only these pieces.
It is clear that after switching to Shamir's scheme, the logic for generating and changing keys appears in the service. To generate the key, a separate virtual machine is used, on which:
As a result, no one can find out the original key, because it is created in the presence of security officers, on a rapidly dying system, and then only "generated" keys are distributed.
When changing keys, it turns out that we can simultaneously have two actual keys in the system: one old, one new, part of the data is encrypted with the old one, and a re-encryption procedure is needed with a new key.
Since it now takes two or three people to run a component, it takes several minutes instead of 30 seconds. Therefore, a simple component after restarting will take several minutes, and you have to switch to an Active-Active scheme, with several simultaneously running instances.
Thus, a simple, obvious service of several tens of lines becomes a rather complex structure: with complex startup logic, clustering, and rather complex maintenance instructions. We happily moved from the normal, simple web to entreprise. And, unfortunately, this happens quite often - much more often than we would like. Moreover, the top management and business, having looked at the whole thing, said that now all data must be encrypted in about the same way, just in case, and he also likes Active-Active everywhere. And these desires of the business, frankly, are not always easy to realize.
Reliable means that we always know whose side the money is on, and if everything has fallen for us now, from whom to demand this money. They can hang with any of the counterparties, the main thing is not with us. And we need to know exactly who they are hung with so that all this can be confirmed and reported to the user. And, of course, it is desirable that there were as few problems as possible.
The first problem is that we have simultaneous events.
We handle an event related, for example, to a user confirming the start of a payment. At this time, an event comes from the counterparty, canceling the possibility of carrying out the operation, and this event also needs to be processed. Therefore, in the logic of work, we have some kind of locks on resources, waiting for the release of locks, etc. Fortunately, in the beginning, all payment processing took place on the same machine and locks could be implemented at the JVM level.
In addition, many steps have a clear maximum execution time (timeout), and these times also need to be stored somewhere, processed, watched when timeout events occur (and they are also sometimes simultaneous).
All this was implemented through the locking logic inside the Java machine, because it was not very easy to do in the database. As a result, we ended up with a system with high availability only through Active-Standby and with a bunch of special logic for restoring contexts and timeouts.
We have a fairly small load, only dozens of payments per second, even less than a hundred in case of the maximum potential peak. At the same time, however, even ten payments per second leads to hundreds of requests (individual steps) per second. These are small loads, so one car is almost always enough for us.
Everything was great, but Active-Active was required.
Making distributed locks is sad, making distributed timeouts also sad. And we began to understand once again - what is a payment? A payment is a set of events that must be processed strictly sequentially, this is a complex mutable state, and payment processing must go in parallel.
Who knew the definition? That's right, payment is an actor.
There are many different actor models in Java. There is a beautiful Akka, there is at times a strange but cool Vert.x, there is a much less used Quasar . They're all great, but they have one fundamental flaw (and not the one you thought) - they lack guarantees.
None of them guarantee the delivery of messages between actors, all of them have a problem with working inside a transaction in the database.
We looked at this for a long time, thought, if we could finish something to a sane state, but then we made our own bike: a queue in PostgreSQL via select for update skip locked.
The whole solution fit into a thousand lines of code and took about two man-weeks for development and two man-weeks for testing and debugging. At the same time, many of our internal needs, which in the same Akka cannot be done normally, were fulfilled.
Suppose we have two tables: a table with our actors - flow, and a table of events for these actors, it is linked by the flow column. Events are sorted by auto-incrementing key ID, everything is normal. We write a SQL query.
We select the very first event in the very first of the flow, specifying the magic for update skip locked. If there are no locks in the table, the request works exactly like a normal for update - it takes and puts a lock on the first line that we have selected, i.e. to the line with the first actor and to the line with the first event for this actor.
We run the same query a second time and it does exactly the same thing, but skips the already blocked lines. Therefore, it will select the first event in the second actor (the third row in the table) and put a lock on it.
Suppose during this time we have finished processing the first of the events, deleted it and closed the transaction. The lock has been removed, so the next time we execute the request, we will receive the first, at the moment, event in the first actor.
This all works fairly quickly and reliably. On cheap hardware, we received about 1000 such operations per second, provided that each of them slows down for about 10 milliseconds. I have used a similar approach several times, all the code is written literally in three lines and it is very easy to attach all sorts of convenient things to such a queue.
You do not have to think about sending messages that cancel the previous ones, do not think about the fact that all messages should be sent in a batch only at the end of processing and after the commit. In general, you stop thinking about many things. For example, you don't need to think about locks , because all your events are processed sequentially, for which, in fact, actors were invented.
In our implementation, we also added a complex error handling policy , because 80% of the payment logic - this is actually the processing of possible errors: the user has gone somewhere, the counterparty answered with some kind of nonsense, the user has no money at all, or the counterparty does not work and it is necessary to choose another counterparty, another gateway, and so on. There is an insane amount of different complex logic for handling all kinds of errors.
This solution is effective for us - 100 payments per second suits us.
But this solution is of very limited applicability - your own bike, which can be used in quite a few places. And it has very tight performance limits. That is, I would not recommend this to colleagues from Yandex.Money, because they have Black Fridays, and 100 payments per second is clearly not enough for them. Fortunately, we don't have Black Friday, we have a very specific market, and therefore we can calmly do with such a decision. At the same time, this is an honest bike, an honest enterprise approach - OpenSource libraries are not very suitable for us in this case.
A gateway is an implementation of a protocol for interacting with one of the money providers. Well, it fell and fell, users did not notice anything, we switched to a backup with another counterparty and began to figure out why. It turned out that the connections in the pool ran out . The reason is not clear, it seems that the load on the gateway was not so big to exhaust all connections.
We begin to understand and find out: our counterparty began to respond to network requests not in half a second, as it was before, but in one minute. Since our processing of a request to a counterparty is a payment step, it is performed in a transaction. When many transactions start to execute for a long time, then there are not enough connections to the database for all incoming requests. This is normal behavior: when you have a lot of long transactions , for some reason your connections start to end.
We began to think what I should do with this. The very first thing is that you can increase the number of connections . Unfortunately, PostgreSQL has clear limits on the maximum number of connections per core., and it is not very large - about a hundred. Because with PostgreSQL, I remind you that each connection is one process. But there are still a lot of processes, tens of thousands or hundreds of thousands of cheap ones cannot be done. And if our counterparties start responding once a minute, then more simultaneous transactions may be required.
You can try to make the network call asynchronous , that is, break each step into two. Every time we need to pull someone out of the counterparties, we need to make a call, save the state to the context database, and get an answer from him. Processing the response will go to the same actor, we will raise the state from the database, do something else we need. But at the same time, the number of steps in the payment increases many times over,and our requirement - 100 payments per second - we no longer fit. And the logic of the work is getting complicated.
All that remains is to manage the guarantees of preservation . We don't always need hard transactionality, we don't always need hard recovery on a network call, we can almost always repeat it. Therefore, we do not need to be able to do everything through the database, we need to be able to do some things bypassing the database , bypassing transactions.
Unfortunately, there is no standard solution that would allow fine-tuning of retention guarantees for a specific event. Now I'm trying to write it, but, honestly, it turns out to be a rather non-trivial task to implement skip locked on some Redis on Lua. If I do all this, I will definitely tell you about it.
As a temporary solution, we split the payment process into several separate actors that run on different DBMSs (and on different servers). This allowed us to introduce asynchronous requests where necessary and solve current problems.
But it turns out that it's cheap if you have little data, and fast if you have little data . As soon as our data volume exceeded 1 GB, it turned out that it was being processed for quite a long time, and most importantly, at some point Microsoft changed the terms of use of the service and it became very paid just starting from about 1 GB. And it turned out that we could not afford it.
Let's go see what we can do.
We began to find out what the business is dreaming of. Business dreams of something like Tableau, where everything is beautiful. Tableau integrates best with Vertica, and it turns out to be an excellent, in theory, system: we throw all events into Kafka, with Kafka we throw them into Vertica.
Vertica works quickly, efficiently, reliably, simply , and Tableau Server shows it all. One but - the cost of the Vertica license is not officially reported, but, to put it mildly, considerable. Tableau isn't very cheap either. Fortunately, it turned out that on our volumes all this is actually not so expensive, because up to one terabyte of data Vertica is free, the Community Edition is absolutely fine for us, we are still far from a terabyte. And since we only need a license for Tableau for a small number of developers and top managers, it costs quite normal money. To the extent that we needed fewer licenses than the minimal package that Tableau sells.
It turned out that such a normal, completely classic heavy enterprise solution is also normal.web solution . It is inexpensive and is built from scratch without hesitation. Vertica still makes me happy: in it, many analytical things are solved very beautifully. Until you have a lot of data, I advise. However, in operation, it is demanding on understanding the principles of its operation, you need to understand them before using.
At the same time, I think that if in a few years we grow beyond the terabyte, then by that time we will have a good expertise on ClickHouse, Tableau will obviously make an adapter for it, and we will neatly crawl to a free ClickHouse for what that's a perfectly reasonable time.
A mistake in this information is quite painful. For example, if we have posted the wrong commissions that we actually charge, users can then be very offended by us, and, most importantly, the regulatory authorities can be offended by us, which is much worse. Therefore, texts for us are also code : we need to check it before publication, many people are involved in its preparation, mistakes are expensive.
At first, our text was just a part of the frontend: all texts were typeset by front-end coders, then went to testing, read, showed on the demo stand, and then went to production. But the text changes too often, and it was simply expensive to do so.
And we began to think about how to automate it alland make CMS. Simple CMSs are not suitable because:
Not simple CMS - too expensive in every sense, they usually cost a lot of money, and their integration is very unclear, because a lot of things have been invented there.
The ideal solution would be to put banal Git for everyone who works with texts : let them send all written texts directly to the repository. But from the idea of putting Git to top managers and a copywriter and teaching them how to use it, we thought and thought and gave up in horror, because after all, git is not for normal people.
The most ideal solution would probably be a text editor built right into IntelliJ IDEA, where you can neatly hide the complexity of using Git. But, unfortunately, JetBrains has not yet made such an editor, although I have asked for them for a long time.
I had to make a bike again:
That is, I had to write, in fact, an enterprise-style micro-portal.
Honestly, if I could buy such a solution, I would rather buy it. But I simply did not find any embedded CMS for large systems on the market, and, in my opinion, they still do not exist. I, frankly, have already written it from scratch for the umpteenth time, and it’s a pity that so far no one has done it for me.
When you have tasks from both the web and the enterprise, you can borrow different ideas from the world of corporations, they have quite a lot of things thought out. Sometimes you can borrow not only ideas, but also specific solutions such as Vertica, if they are cheap.
Honestly, if I found cheap support for IBM DB2 - I would implement a project on it, I love it very much, it is cheap and very reliable, but it is difficult to find support for this database for reasonable money in Russia. Of course, you can lure someone from Russian Post, but they are used to servers so large that we are clearly too small for them.
Well big problems from the enterprise world can be solved in a web-style quite simply, which is what we do all the time.
Architecture is a dynamic concept.
There is no good architecture at all. There is an architecture that more or less satisfies you at a particular point in time . Time is changing - architecture is changing, and we must constantly be ready for this, and always invest resources in the development of architecture. Project architecture is a process, not a result.
Java and SQL are really cool if you know how to cook it. We know how, so everything turns out for us simply, quickly, effortlessly, and with a very small team we make it quite difficult
Under the cut, the story of Philippe Delgado (dph) on Highload ++ about the experience accumulated over several years of working on a payment system for the Russian legal betting business, about mistakes, but also about some achievements, and about how to competently mix, but not shake up, the web with the enterprise.
About the speaker: Philippe Delgado has done a lot of things during his career - from two links in
Visual Basic to hardcore SQL. In recent years, he has been mainly engaged in loaded projects in Java and regularly shares his experience at various conferences.
We have been making our payment system for three years, of which we have been in production for two years. Two years ago I told you how to make a payment system in one year, but since then, of course, a lot has changed in our solution.
We are a rather small team: 10 programmers, mostly back-end developers and only two people on the front-end, four QA and me, plus some kind of management. Since the team is small, there is not much money, especially in the beginning.
Payment system
In general, the payment system is very simple: take money, transfer records from one plate to the same plate with a minus sign - in general, that's all!In reality, this is how it is, the payment system is a very simple thing. Until the lawyers came . Payment systems all over the world are subject to a huge number of all kinds of burdens and instructions on how to transfer money from one tablet to another correctly, how to interact with users, what they can be promised, what cannot be promised, what we are responsible for, what we are not responsible for. Therefore, as part of the development of a payment system, you have to constantly balance on the verge between a heavy enterprise and a completely normal scalable web application.
From the enterprise we have the following.
We work with money.
Therefore, we have complex accounting, we need a high level of reliability, a high SLA (because neither we nor the users like a simple system), and high responsibility - we must know exactly where and what user's money went and what happened to them. the moment generally happens.
We are an NPO (non-bank credit institution).
It is practically a bank, but we cannot issue loans.
- We have reporting to the Central Bank;
- We have reporting to Finmonitoring;
- We have many colleagues in the banking part of the company and with banking experience;
- We are forced to interact with the automated banking system.
We have lawyers. Here are some of the laws that govern the behavior of payment systems at the moment in the Russian Federation:
Each of these laws is quite difficult to implement, because there are still a bunch of bylaws, real use stories, and you have to work with all this.
At the same time, besides the fact that we are such a pure enterprise, almost a bank, we are still quite a web company.
- For us, the convenience of the user is of fundamental importance, because the market is highly competitive, and if we do not take care of our user, he will leave us and we will not have any money at all.
- We are forced to make frequent calculations because the business is actively developing. Now we have releases 23 times a week, which is much more often than those of large banks, where, they say, they have recently finally started making releases once every 3 months and are very proud of it.
- Minimum time-to-market: as soon as an idea comes up, you need to launch it into real life as soon as possible - preferably faster than competitors.
- We don't have a lot of money, unlike many large banks. We cannot take and fill everything with money, we have to somehow get out, make some decisions.
We use Java because it is somewhat easier to find developers on the market who at least roughly understand about the reliability of working with databases in the Java world than in other languages.
I know only three databases with which you can make a payment system, but one of them is very expensive, for the second it is difficult to find support and it is also not free. As a result, PostgreSQL is the best option for us: it's easy to find sane support, and in general, for little money, you don't have to think about what is happening with the databases at all: everything is clean, beautiful and guaranteed.
The project uses a little Kotlin - rather for fun and to look to the future. So far, Kotlin is mostly used as some scripting language, plus some small services. Service architecture
, of course . It is categorically impossible to call it microservices. Microservice, in my understanding, is something that is easier to rewrite than to understand and refactor. Therefore, we, of course, do not have a microservice, but normal full-fledged large components.
In addition, Redis for caching, Angular for backend. The main site, visible to the user, is made on pure HTML + CSS with a minimum of JS.
And of course, Kafka.
Services
Of course, I would rather live without services. There would be one big monolith, no connectivity problems, no versioning problems: take it, write it, put it up. It's simple.But security requirements come . We have personal data in the system, personal data must be stored and processed separately, with special restrictions. We have information on bank cards, it must also live in a separate part of the system with the appropriate audit requirements for each code change and requirements for data access. Therefore, you have to cut everything into components.
There are requirements for reliability... I don’t want to, because some gateway with one of the counterparty banks for some reason broke down, take and lay out all the payment logic: God forbid, there will be an incorrect calculation, the human factor - and everything will collapse. Therefore, you still have to divide everything into relatively small services.
But since we are starting to divide the system into components according to security and reliability requirements , it makes sense to separate out into separate services everything that requires its own storage. Those. for a part of the database that does not depend on the whole other system at all, it is easier to create a separate service.
"Processing" or "Reports" is a more or less normal name, "Bullshit that works with a database replica" is already a bad name. Obviously, this is not one service - either several, or part of one large one.
These four requirements are quite enough for us to allocate separate
services.
Of course, we still have micromonoliths that we keep cutting because they accumulate too many words and too many names for one service. This is an ongoing process of redistribution of responsibility.
The services themselves interact via JSON RPC over http (s), as in any web world. At the same time, for each service, a separate logic for retrying requests and caching results is prescribed. As a result, even if a service crashes, the entire system continues to work normally, and the user does not notice anything.
Components
Kafka
This is not a message queue, our Kafka is only a transport layer between services, with a delivery guarantee and clear reliability / clustering . Those. if you need to send something from service A to service B, it is easier to put a message in Kafku, from Kafk and the service itself will take what is needed. And then you don't have to think about all this logic for retrying and caching, Kafka greatly simplifies this interaction. Now we are trying to translate as much as possible into Kafka, this is also such an ongoing process.Well, among other things, this is a backup source of data about all our operations.... I am, of course, paranoid, since the specifics of the work are conducive. I saw (quite a long time ago and not in this project) how a commercial database for a lot of money at some point began to write nonsense not only to its own database files, but also to all replicas and backups. And the data had to be restored from the logs, because it was data on the payments made, and without them the company could safely close the next day.
I don't really like pulling out important data from the logs, so I'd rather put all the necessary information into the same Kafku. If suddenly some impossible situation happens to me, I, at least, know where to get the backup data from, which has nothing to do with the main storage.
In general, for payment systems to have two independent data storages is a standard practice, living without it is simply scary - for me, for example.
Development logs
Of course, we have a lot of different logs. We are now saving the development logs in Kafku and then uploading them to Clickhouse, because, as it turned out, it's easier and cheaper. Moreover, at the same time we are studying Clickhouse, which is useful for the future. However, you can make a separate report about working with logs.Monitoring
We have monitoring at Prometheus + Grafana. Honestly, I'm not happy with Prometheu.What is the problem?
- Prometheus is great when you need to collect some data from ready-made standard components, and you have a lot of these components. We have quite a few cars. We have 40 different services and that's about 150 virtual machines, that's not a lot. If we want to collect some kind of business monitoring information through Prometheus, for example, the number of payments going through a certain gateway, or the number of events in our internal queue, then we have to write quite a lot of code on the client side. Moreover, the code, unfortunately, is not very simple, developers have to actively understand the internal logic and how exactly Prometheus thinks something.
- Prometheus cannot be used as an honest event oriented time-series db. I cannot take and say that there is a payment start event, a payment end event, and let him calculate all other metrics. I have to calculate all the metrics I need on the client in advance, and if suddenly I need to change any of them, this is another calculation of the production component, which is very inconvenient.
- It is very difficult to do integrated metrics. If I need to collect a common metric for a certain number of services (for example, percentiles of the response time to clients across all front-end servers), then it is unrealistic to do this through Prometheus, even theoretically. I can only do some kind of incomprehensible average summation already at the Grafana level. Prometheus itself cannot do this.
Therefore, I seriously think to go somewhere.
Further, I will tell several separate cases about what architectural challenges we had, how we solved them, what was good and what was bad at the same time.
Database usage
In general, payment is quite difficult. Below is an approximate description of the payment context: a set of tuples (associative arrays), lists of tuples, tuples of lists, some parameters. And all of this is constantly changing due to changes in business logic.To be honest, there will be many tables, many relationships between them. As a result, you need an ORM, you need complex migration logic when adding a column. Let me remind you that in PostgreSQL, even a simple addition of a new nullable column to a table can lead (in some specific situations) to the fact that for a long time this table will not be available at all. Those. in fact, adding a nullable column is not an atomic free operation, as many people think. We even stumbled upon it once.
All this is rather unpleasant and sad, I want to avoid all this, especially when using ORM. Therefore, we remove all these large and complex entities in JSON, simply because in reality, except on the application server, all these data and structures are not needed anywhere. I have been using this approach for 10 years and, finally, I notice that it is becoming, if not mainstream, then at least a generally accepted practice.
JSON Practices
As a rule of thumb, storing complex business data in a database as JSON will not lose anything in terms of performance, and maybe even gain at times. Then I will tell you how to do this so as not to accidentally shoot yourself in the leg.First, you must immediately think about possible conflicts.
Once you had a version of an object with one set of data fields, you released another version, where there is already a different set of data fields, you need to somehow read the old JSON and convert it into an object convenient for you.
To solve this problem, it is usually enough to find a good serializer / deserializer, to which you can explicitly tell that this field from JSON needs to be converted into such and such a set of fields, these things should be serialized this way, and if something is not there, then replace with default, etc. In Java, fortunately, there are no problems with such serializers. My favorite is Jackson.
Be sure to store the version of the structure you are writing in the database.
Those. next to each field where you store your JSON, there should be another field where the version is stored. First of all, this is necessary in order not to endlessly support the code of understanding the old version of the new one.
When you release a new version and you have a new data structure, you just make a migration script that runs through the entire database, finds all the old versions of the structure, reads them, writes them in the new format, and after some rather limited time you have , a maximum of 2-3 different versions of the data remains in the database, and you do not bother with the support of all the diversity that you have accumulated over the years. It is ridding yourself of legacy, ridding yourself of technical debt.
For PostgreSQL, you have to choose between json and jsonb.
Once upon a time, this choice still made sense. For example, we used JSON because we started a long time ago. Let me remind you that the JSON data type is just a text field, and PostgreSQL will parse it every time to get inside. Therefore, in production it is better not to get inside the json objects in the database once again, only in case of some kind of support or troubleshooting. In an amicable way, your SQL code should not contain commands for working with json fields at all.
If you use JSONB, then PostgreSQL neatly parses everything into a binary format, but does not preserve the original form of the JSON object. When we, for example, store the original data coming to us, we always use only JSON.
We don't need JSONB yet, but at the moment, it really makes sense to always use JSONB and not think about it. The difference in performance has become practically zero, even for simple read and write.
PCI DSS. From simple to complex, and how a web becomes an entrepreneur
Even at the development stage, long before going into production, we had a small simple service with bank card data, including the card number itself, which we, of course, encrypted using PostgreSQL . At the same time, theoretically, the operation manager could, probably, find somewhere the key of this encryption and find out something, but we completely trusted him .The reliability of the service was realized through active-standby - because the service is small, it restarts quickly, other components will wait for 3-5 seconds, so there is no point in piling up some complex cluster system.
Before launching, we began to undergo a PCI DSS audit, and it turned out that there are rather strict requirements for data access control, which, in the case of our auditor, boiled down to the following:
- There should not be one person who can read all the information from the database. There must be at least several people who must jointly get access.
- Regular change of access keys is required.
- PCI DSS requires the infrastructure to be updated for any discovered vulnerability, and since vulnerabilities in the operating system and infrastructure software are found quite often, the system must also be updated quite often.
To begin with, we stop trusting the operation manager and try to come up with a scheme when we do not have one person who knows the keys.
Logically, we come to Shamir's scheme. This is a way of generating a key, when several keys are generated based on the ready key, any subset of which can generate the original key.
For example, you form a long key, immediately break it into 5 pieces so that any three of them could generate the original one. Then you distribute these three exploits, store two in a safe, just in case someone gets sick, gets hit by a bus, etc., and live peacefully. You no longer need the original long key, only these pieces.
It is clear that after switching to Shamir's scheme, the logic for generating and changing keys appears in the service. To generate the key, a separate virtual machine is used, on which:
- a key is generated,
- distributed to admins,
- the virtual girl is killed.
As a result, no one can find out the original key, because it is created in the presence of security officers, on a rapidly dying system, and then only "generated" keys are distributed.
When changing keys, it turns out that we can simultaneously have two actual keys in the system: one old, one new, part of the data is encrypted with the old one, and a re-encryption procedure is needed with a new key.
Since it now takes two or three people to run a component, it takes several minutes instead of 30 seconds. Therefore, a simple component after restarting will take several minutes, and you have to switch to an Active-Active scheme, with several simultaneously running instances.
Thus, a simple, obvious service of several tens of lines becomes a rather complex structure: with complex startup logic, clustering, and rather complex maintenance instructions. We happily moved from the normal, simple web to entreprise. And, unfortunately, this happens quite often - much more often than we would like. Moreover, the top management and business, having looked at the whole thing, said that now all data must be encrypted in about the same way, just in case, and he also likes Active-Active everywhere. And these desires of the business, frankly, are not always easy to realize.
Payment logic. Make simple from complex
As I said before, payment is quite difficult. Below is an approximate diagram of the process of transferring money from the user to the final counterparty, but not everything is shown in the diagram. In the process of payment, there are many dependencies on some external entities: there are banks, there are counterparties, there is a banking information system, there are transactions, postings, and all this should work reliably.Reliable means that we always know whose side the money is on, and if everything has fallen for us now, from whom to demand this money. They can hang with any of the counterparties, the main thing is not with us. And we need to know exactly who they are hung with so that all this can be confirmed and reported to the user. And, of course, it is desirable that there were as few problems as possible.
Finite State Machine
Of course, in the beginning we had an FSM - a normal state machine, each event is processed in a transaction. We also save the current state in the DBMS. We did it all ourselves.The first problem is that we have simultaneous events.
We handle an event related, for example, to a user confirming the start of a payment. At this time, an event comes from the counterparty, canceling the possibility of carrying out the operation, and this event also needs to be processed. Therefore, in the logic of work, we have some kind of locks on resources, waiting for the release of locks, etc. Fortunately, in the beginning, all payment processing took place on the same machine and locks could be implemented at the JVM level.
In addition, many steps have a clear maximum execution time (timeout), and these times also need to be stored somewhere, processed, watched when timeout events occur (and they are also sometimes simultaneous).
All this was implemented through the locking logic inside the Java machine, because it was not very easy to do in the database. As a result, we ended up with a system with high availability only through Active-Standby and with a bunch of special logic for restoring contexts and timeouts.
We have a fairly small load, only dozens of payments per second, even less than a hundred in case of the maximum potential peak. At the same time, however, even ten payments per second leads to hundreds of requests (individual steps) per second. These are small loads, so one car is almost always enough for us.
Everything was great, but Active-Active was required.
Active-Active
Firstly, we wanted to use Shamir's scheme, and other wishes also appeared: let's publish the new version only to 3% of users; let's change the logic of the payment often; I want to lay it out with zero downtime, etc.Making distributed locks is sad, making distributed timeouts also sad. And we began to understand once again - what is a payment? A payment is a set of events that must be processed strictly sequentially, this is a complex mutable state, and payment processing must go in parallel.
Who knew the definition? That's right, payment is an actor.
There are many different actor models in Java. There is a beautiful Akka, there is at times a strange but cool Vert.x, there is a much less used Quasar . They're all great, but they have one fundamental flaw (and not the one you thought) - they lack guarantees.
None of them guarantee the delivery of messages between actors, all of them have a problem with working inside a transaction in the database.
We looked at this for a long time, thought, if we could finish something to a sane state, but then we made our own bike: a queue in PostgreSQL via select for update skip locked.
The whole solution fit into a thousand lines of code and took about two man-weeks for development and two man-weeks for testing and debugging. At the same time, many of our internal needs, which in the same Akka cannot be done normally, were fulfilled.
Skip locked
This is such a great thing for implementing queues in PostgreSQL. In fact, all databases have this mechanism, except, in my opinion, MySQL.Suppose we have two tables: a table with our actors - flow, and a table of events for these actors, it is linked by the flow column. Events are sorted by auto-incrementing key ID, everything is normal. We write a SQL query.
We select the very first event in the very first of the flow, specifying the magic for update skip locked. If there are no locks in the table, the request works exactly like a normal for update - it takes and puts a lock on the first line that we have selected, i.e. to the line with the first actor and to the line with the first event for this actor.
We run the same query a second time and it does exactly the same thing, but skips the already blocked lines. Therefore, it will select the first event in the second actor (the third row in the table) and put a lock on it.
Suppose during this time we have finished processing the first of the events, deleted it and closed the transaction. The lock has been removed, so the next time we execute the request, we will receive the first, at the moment, event in the first actor.
This all works fairly quickly and reliably. On cheap hardware, we received about 1000 such operations per second, provided that each of them slows down for about 10 milliseconds. I have used a similar approach several times, all the code is written literally in three lines and it is very easy to attach all sorts of convenient things to such a queue.
What do we get with such a queue?
All messages are transactional: we started a transaction, in it we do something with the database, in it we send messages to other actors somewhere, if the transaction is rolled back, messages will not be sent either, which is insanely convenient.You do not have to think about sending messages that cancel the previous ones, do not think about the fact that all messages should be sent in a batch only at the end of processing and after the commit. In general, you stop thinking about many things. For example, you don't need to think about locks , because all your events are processed sequentially, for which, in fact, actors were invented.
In our implementation, we also added a complex error handling policy , because 80% of the payment logic - this is actually the processing of possible errors: the user has gone somewhere, the counterparty answered with some kind of nonsense, the user has no money at all, or the counterparty does not work and it is necessary to choose another counterparty, another gateway, and so on. There is an insane amount of different complex logic for handling all kinds of errors.
This solution is effective for us - 100 payments per second suits us.
But this solution is of very limited applicability - your own bike, which can be used in quite a few places. And it has very tight performance limits. That is, I would not recommend this to colleagues from Yandex.Money, because they have Black Fridays, and 100 payments per second is clearly not enough for them. Fortunately, we don't have Black Friday, we have a very specific market, and therefore we can calmly do with such a decision. At the same time, this is an honest bike, an honest enterprise approach - OpenSource libraries are not very suitable for us in this case.
Network and transactions
Everything went smoothly on paper. We implemented it, launched it - it works. And suddenly a problem comes - one of the locks has fallen .A gateway is an implementation of a protocol for interacting with one of the money providers. Well, it fell and fell, users did not notice anything, we switched to a backup with another counterparty and began to figure out why. It turned out that the connections in the pool ran out . The reason is not clear, it seems that the load on the gateway was not so big to exhaust all connections.
We begin to understand and find out: our counterparty began to respond to network requests not in half a second, as it was before, but in one minute. Since our processing of a request to a counterparty is a payment step, it is performed in a transaction. When many transactions start to execute for a long time, then there are not enough connections to the database for all incoming requests. This is normal behavior: when you have a lot of long transactions , for some reason your connections start to end.
We began to think what I should do with this. The very first thing is that you can increase the number of connections . Unfortunately, PostgreSQL has clear limits on the maximum number of connections per core., and it is not very large - about a hundred. Because with PostgreSQL, I remind you that each connection is one process. But there are still a lot of processes, tens of thousands or hundreds of thousands of cheap ones cannot be done. And if our counterparties start responding once a minute, then more simultaneous transactions may be required.
You can try to make the network call asynchronous , that is, break each step into two. Every time we need to pull someone out of the counterparties, we need to make a call, save the state to the context database, and get an answer from him. Processing the response will go to the same actor, we will raise the state from the database, do something else we need. But at the same time, the number of steps in the payment increases many times over,and our requirement - 100 payments per second - we no longer fit. And the logic of the work is getting complicated.
All that remains is to manage the guarantees of preservation . We don't always need hard transactionality, we don't always need hard recovery on a network call, we can almost always repeat it. Therefore, we do not need to be able to do everything through the database, we need to be able to do some things bypassing the database , bypassing transactions.
Unfortunately, there is no standard solution that would allow fine-tuning of retention guarantees for a specific event. Now I'm trying to write it, but, honestly, it turns out to be a rather non-trivial task to implement skip locked on some Redis on Lua. If I do all this, I will definitely tell you about it.
As a temporary solution, we split the payment process into several separate actors that run on different DBMSs (and on different servers). This allowed us to introduce asynchronous requests where necessary and solve current problems.
Accounting and control. Budget Business Intelligence
We have a payment system, that is, money, and money is loved when it is counted. Therefore, a business came to us very quickly with a request to make a Business Intelligence system. We do n't have a lot of data , some hundreds of gigabytes, only top management needs it, we don't have hundreds of analysts. And the main thing is to do it “ quickly and cheaply ”.Power BI - Fast and Cheap?
We take PowerBI - this is a solution from Microsoft: the system generates the necessary data in the form of csv, csv is uploaded to the cloud, from the cloud they are uploaded to PowerBI. Cheap, fast, simple, made literally on the knee, almost completely without the involvement of programmers. It is not difficult to write reports in csv.But it turns out that it's cheap if you have little data, and fast if you have little data . As soon as our data volume exceeded 1 GB, it turned out that it was being processed for quite a long time, and most importantly, at some point Microsoft changed the terms of use of the service and it became very paid just starting from about 1 GB. And it turned out that we could not afford it.
Let's go see what we can do.
ClickHouse
First thought - hurray, there is ClickHouse! We throw all our events into Kafku, from there we upload them in batches to ClickHouse, it turns out cool, fashionable, hype, analytics should work quickly, everything should be free, generally fine and wonderful. But the result from ClickHouse must be shown somewhere. At the moment, Redash works best with Clickhouse. We made a test version of Redash, showed it to the business - they said that they would not work with it, because it looks, to put it mildly, ugly and some nice drill-down things for business are simply not there.We began to find out what the business is dreaming of. Business dreams of something like Tableau, where everything is beautiful. Tableau integrates best with Vertica, and it turns out to be an excellent, in theory, system: we throw all events into Kafka, with Kafka we throw them into Vertica.
Vertica works quickly, efficiently, reliably, simply , and Tableau Server shows it all. One but - the cost of the Vertica license is not officially reported, but, to put it mildly, considerable. Tableau isn't very cheap either. Fortunately, it turned out that on our volumes all this is actually not so expensive, because up to one terabyte of data Vertica is free, the Community Edition is absolutely fine for us, we are still far from a terabyte. And since we only need a license for Tableau for a small number of developers and top managers, it costs quite normal money. To the extent that we needed fewer licenses than the minimal package that Tableau sells.
It turned out that such a normal, completely classic heavy enterprise solution is also normal.web solution . It is inexpensive and is built from scratch without hesitation. Vertica still makes me happy: in it, many analytical things are solved very beautifully. Until you have a lot of data, I advise. However, in operation, it is demanding on understanding the principles of its operation, you need to understand them before using.
At the same time, I think that if in a few years we grow beyond the terabyte, then by that time we will have a good expertise on ClickHouse, Tableau will obviously make an adapter for it, and we will neatly crawl to a free ClickHouse for what that's a perfectly reasonable time.
Content
We have quite a few texts:- Legal information that we are required to provide, offers, etc .;
- Information about counterparties, including commissions that we charge from users;
- User instructions;
- Own small blog;
- Error information and more.
A mistake in this information is quite painful. For example, if we have posted the wrong commissions that we actually charge, users can then be very offended by us, and, most importantly, the regulatory authorities can be offended by us, which is much worse. Therefore, texts for us are also code : we need to check it before publication, many people are involved in its preparation, mistakes are expensive.
At first, our text was just a part of the frontend: all texts were typeset by front-end coders, then went to testing, read, showed on the demo stand, and then went to production. But the text changes too often, and it was simply expensive to do so.
And we began to think about how to automate it alland make CMS. Simple CMSs are not suitable because:
- difficult to separate the test environment and production;
- it is not clear how to test the text;
- it is difficult for a large number of users to work;
- difficult to integrate with a large Java system.
Not simple CMS - too expensive in every sense, they usually cost a lot of money, and their integration is very unclear, because a lot of things have been invented there.
The ideal solution would be to put banal Git for everyone who works with texts : let them send all written texts directly to the repository. But from the idea of putting Git to top managers and a copywriter and teaching them how to use it, we thought and thought and gave up in horror, because after all, git is not for normal people.
The most ideal solution would probably be a text editor built right into IntelliJ IDEA, where you can neatly hide the complexity of using Git. But, unfortunately, JetBrains has not yet made such an editor, although I have asked for them for a long time.
I had to make a bike again:
- Simple text editor.
- A simple html editor, because instead of a complex text editor, it is easier to ask our front-end developer to typeset everything in html, but at the same time upload it through the CMS and quality control system.
- A simple concept of versions - there are packages of changes, the publication is entirely in packages, restoration, if necessary, of all content is also one button at a time. And a very simple procedure for working with texts.
- A categorical prohibition (there is simply no such button) to change anything directly in production. Everything can be changed only through a task in Jira, then it goes into the test version, separately trained people are already transferring from the test version to production - all they can do is transfer ready-made packages from the test version to production, after all confirmations have been received. without the ability to make edits.
That is, I had to write, in fact, an enterprise-style micro-portal.
Honestly, if I could buy such a solution, I would rather buy it. But I simply did not find any embedded CMS for large systems on the market, and, in my opinion, they still do not exist. I, frankly, have already written it from scratch for the umpteenth time, and it’s a pity that so far no one has done it for me.
Conclusions
What of all this can be said? That life on the edge is pretty interesting.When you have tasks from both the web and the enterprise, you can borrow different ideas from the world of corporations, they have quite a lot of things thought out. Sometimes you can borrow not only ideas, but also specific solutions such as Vertica, if they are cheap.
Honestly, if I found cheap support for IBM DB2 - I would implement a project on it, I love it very much, it is cheap and very reliable, but it is difficult to find support for this database for reasonable money in Russia. Of course, you can lure someone from Russian Post, but they are used to servers so large that we are clearly too small for them.
Well big problems from the enterprise world can be solved in a web-style quite simply, which is what we do all the time.
Architecture is a dynamic concept.
There is no good architecture at all. There is an architecture that more or less satisfies you at a particular point in time . Time is changing - architecture is changing, and we must constantly be ready for this, and always invest resources in the development of architecture. Project architecture is a process, not a result.
Java and SQL are really cool if you know how to cook it. We know how, so everything turns out for us simply, quickly, effortlessly, and with a very small team we make it quite difficult
