How the DGB.LOL website works?
Published: 16.09.2023 - Tom
This website is one of my biggest (or even THE biggest) projects in my life. To bringing the website to its current shape brought me a lot of sleepless nights, solving thousands, if not millions of small, bigger or huge problems, integrating completely new technologies including AI, putting them together with my old knowlegde, improving my old knowledge and reading thousands of pages of documentation.
In the older version of the website I've also faced a bigger hacker attack, when some of Bangladeshi "hackers" tried to misuse my service and make a profit for himself. This time I was on my vacation with extremely limited (and expensive) Internet access. But this is not a story about that, I just wanted to mention the variety of problems I've faced and needed to resolve...
In this article, I want to give you a closer look "behind the scenes" of the technology and processes I am using to make from your 1024x1024 pixels image a HD masterpiece, and more. Keep reading, I will try to explain you how it works "under the hood" of the dgb.lol's website!
Have you ever wondered, what is the journey of your image from the upload button to the download phase?
I was thinking a lot, how to explain to you this complex and complicated technology I am using on the website, to the easiest, simplest way possible.
Nothing came to my mind, until I went to the train station, and then boom! got it! I will explain it to you step by step with an analogy with a train station.
So imagine the following: the dgb.lol website is a big train station, each visitor of the website is a passenger that wants to travel (use the tools). There are a lots of destinations (analogy = tools) , but only 4 rails (servers) leading to the destination (process his images/files by using the dgb.lol's tools).
Currently, there are 4 trains, leading to 10 destinations (tools)
The CPU train
This train is bringing its passengers to the following destinations:
- Quick Upscaler
- Background Remover
- RGB to CMYK
- DPI Enhancer
- Image Converter
- SVG to EPS Converter
This train has only 1 seat, it means that only one user can travel to its destination at the time. The others are waiting in the queue.
The advantage of this train is, that it's pretty quick, every destination is being reached within few seconds, in worst case scenario, in a minute.
This platform is also the less occupied, not that much passengers are travelling (using the tools) with that train. The daily average is around 900-1100 users being transported (served)
These passengers are travelling a single route:
1) Uploading the file to the server
2) Waiting for the short queue
3) The file is already at the location - the same server handles the processing as the files are being uploaded - it means the dgb webserver has a lot of CPUs and it is able to process these files without slowing down the website.
4) The file is being processed and the download link generated
5) That's it, the journey is over :)
The TESLA train complex
Here, the things are being a bit more complicated, however this train has only 2 destinations:
- AI Image Upscaler
- Midjourney Splitter
This train has 5 seats, and running paralelly from two platforms.
It means, that both of the trains are going from the platform to the destination and back with 5 passengers at the same time, as they have the same destination. The same destination means, that the same process handling the Midjourney images and the custom images uploaded by the users. There is no difference, both images are simply - images :)
When the user uploads and submits it's image for upscaling/MJ splitting, he gets a ticket at the same moment, so he knows where to wait, platform 1, or platform 2 (dgb.lol has 2 upscaler servers, also called clusters).
The dgb.lol web server knows how many passengers are waiting in the queues on both platforms. Once you upload the image, you are getting a train ticket for the platform with the less users waiting in the queue.
Once the train arrives, and you are between the 5 lucky passengers who is getting a ride (you are in front of the queue), you (your image) are travelling to the cloud. To the Google cloud server on a secure, but longer way. It means, that your original image is being transferred to the Google Cloud Compute server, it brings with himself a train ticket (upscaler parameters like model, dimensions, task number, owner - you, etc..)
The Cloud server has really powerful graphic cards, called Nvidia Tesla. They are working all day at full throttle to process the files based on their parameters, by using advanced AI technology - called Real ESRGAN. I will not complicate more the things, this is a very complex thing and even the smaller image takes 10-30 seconds to be processed with the AI and upscaled to its best possible shape!
So, once all of the 5 passengers have reached their destination (been upscaled), they can return back from the cloud, back to the homeland - the dgb.lol web server.
They are using a fully secure way to go back from the cloud to the server, and based on their ticket, the dgb.lol web server knows their owner, their name, their task number and so on.
Once it's finished, all of the 5 passengers are getting a download link and the status "finished". These passengers are going out from the train, and immediately the next 5 lucky passengers are sitting to that train and the journey starts again!
The whole journey takes from 1 to 3 minutes for these 5 passengers.
This train station is the most over crowded place on the dgb.lol's train station.
These two upscaler platforms are non stop occupied, and the current train power handles over 12000 passengers every single day! The supervisor (me-Tom) is checking the trains performance, the need for maintenance multiple times per day.
If the supervisor thinks, that the crowd is too huge, he installs more graphic cards to the cloud server.
The Premium TESLA train
This is the newest platform in the dgb.lol train station, it has two destinations:
- AI Image Upscaler
- Midjourney Splitter
There are only few differences in comparsion with the TESLA train complex:
- It has only one platform
- It can hold up to 15 passengers
- Travelling 15x faster than the original TESLA train.
- The queue is almost non existant
- These train tickets are being paid with real money (Premium credits)
The rest is pretty much the same as for the TESLA train complex.
When there is a huge queue in the non-premium TESLA train platforms, and the premium train platform is empty, or it has no more than 5 passengers, it accepts 3-4 normal passengers, only to reduce the waiting time of the normal passengers, but it never forces to wait the premium passengers.
I really hope that the idea and the analogy with the train station was a good match and you have better insight, what is happening "under the hood" while you are waiting to your image(s) to be processed.
Believe me, it's easy to read, but much more complicated to melt it down into the code, even harder to convert these processes to the code without bugs :)