This article is provided by mcrdy455 from earth2.market - he is a BOTs developer for a living and knows a thing or two about stopping bots. This is part two of a two part article. Read Part One.
Author's Note
Make sure you read to the end, because there I’ve also shared some tips on a way to stop services like earth2.market/premium
So… What’s the solution...
I’ll attach another screenshot, with the list of solutions I’ve suggested in the past:
If my messages were to be answered I’d have shared all my knowledge on how to STOP or Slow Down bots. Keep in mind that I develop bots for a living: I maintain bots for Facebook, Google, and other top websites, which unlike earth2.io bots, have personality…
What happened after that?
I’ve tried contacting many staff members on the matter, but the only one who answered for some time was Thomas and he recently stopped answering as well… Over the last month, services like earth2.market/premium started popping up left and right - this resulted in me being forced to reduce the refresh rate from 180 seconds to 10 seconds for a full cycle, which actually is 100 times more pressure on earth2.io servers, because its not only me doing it. The difference is that earth2.market is used by 400-800 unique users daily, compared to some scrapers which are used by a single user. Also earth2.market has a FREE public market API which is used by websites like earth2stats.xyz to get the marketplace data, without the need of scraping earth2.io… Why is Earth2.io having so hard time with adding new features and what are my proposed solutions..
The Earth 2 team, have shared that they experienced unexpected growth. Which resulted in many delays. Which as a developer I understand, but also: Growth = Money. Money = Hire professionals..
Where is the problem in that? The answer is simple: Planning... I can bet that they haven’t done their software architecture when they had to, and now they want to build on top of a really weak base. And if you build a skyscraper on a weak base… It will collapse.
Here is the proof for my words:
Why do you think the earth2.market has so many good filters?
The answer is because I have a much smaller database and I store very little data in it, which allows me to have much more flexible filters. If you take a look at their database, you can see they store some data, which they don't use for the marketplace - like the last time a property has been modified, the time when a property has been put for sale, there is also some data about the Land Income Tax, which I’m 100% sure, there is no point storing together with the data for the marketplace. And the result was seen recently when they added a query to get all marketplace properties… and for some reason they had to revert those changes in less than a day… (probably due to architectural mistake, or the servers overloading)
My suggestion is probably something that they have already implemented: Store properties for sale in a separate database, containing only data such as price, discount and other market related stuff. And remove those database properties from the main database
"The Mystical Eye - Ion Mystical World" by Daniel Arrhakis is licensed with CC BY-NC 2.0.
Do earth2 developers have a plan?
Since creating the earth2.market website, the API endpoints, have changed multiple times. Which is a sign of fixing a mistake… In my opinion, moving away from GraphQL for the market requests was another mistake, since I used to send very few requests to update my database, with smart GraphQL queries but now I have to spam requests...
Another thing I noticed was that they are adding new data properties to the the land fields database, which is BAD, because it adds data exponentially. Adding even a single Boolean property (which has a value or 0 or 1) can cause huge database size increase, (multiply 1 byte by 3million land fields...) Its hard to do operations with huge databases.
Seeing the problems as they come and how long it takes to solve them my suggestion here is to do what has always been done in such situation. Rebuild the project from scratch… I don’t know of a big project which hasn’t been rebuilt from scratch at least once (Facebook has completely been rebuilt just in the last year). Now that you know your obstacles and weaknesses, you can learn from your mistakes and improve.
Why so many promises, so many secrets, so silent?
We were told, there will be no problems on the UAE launch… They were wrong… Why can’t you just admit that “there might be some issues”. No matter when they decide to rewrite the entire project from scratch, they will have to announce it.. Or there will just be a long stretch of time with no updates (they may be rewriting the project from scratch right now… Which I’d be happy if they shared with us...)
I’ve seen many of the most active players getting their accounts frozen without any information for weeks… I know why they are being frozen, but it’s a secret shared with very few players, and the rest are just wondering... not a single announcement, or anything - the reason I was given for that is to prevent players from thinking there is any problem with the project. (THERE ARE MANY PROBLEMS WITH EVERY PROJECT WTF...)
Honesty is what keeps the players united, and your supporters . So that's what I’d suggest. While writing this article I asked many users for advice, one of them shared the AfterEarth project, where the devs listen to the community and fix bugs fast… What earth2.market competitors have taught me is how to improve, I used them as motivation. I’ve raced to be the best service, I’ve raced the be the biggest and the most accessible. What is earth2.io doing with competitors?
You shouldn’t remain silent… that will lead to your failure…
Finally, about bots…
Here is auto-buy script for buying new land in earth2: (could have been used by many users on UAE launch)
Back then you didn’t need captcha solving service... Worst thing is that it works… What happens when you run similar code on earth2.market/premium?
Nothing… You get a simple alert and you’ll be slowed down… (the solution is a simple line of code) This is the reason many users moved from earth2.market/premium to other paid services… Its a bit harder building bots on a website created by a bot developer, but still…
You CAN NOT stop bots… all you can and should do is slow them down while giving the humans the advantage… The worst thing is that misinformed decisions on the earth2.io platform gave the bots the advantage.
Another thing you need to consider - external services?
Talking sh*t about “going to stop them all” is a really bad joke.. imagine a world where all the services provided were provided by only 1 corporation? Is Earth2 a communist project? Once you start adding more and more functionality to the game, it's natural that people providing some services will emerge. Doesn’t matter if its architectural, market, advisory, or withdrawal services..
I suggest you even add a PTP service, where people can exchange E$ for PayPal money or CryptoCurrency. You should have the option to donate money and properties… The more actions you move into players hands the faster earth2.io will grow...
How to stop market sniper services?
I’m not worried about having my website rendered useless… I’ll be happy!!! I’ll always find something else to do.
I have domains like earth2.lol, earth2.party or the project I’m currently working on: earth2.auction, which I’ve put on hold and I’m afraid to work on, because earth2.io is SILENT…
It's very simple to stop them. Its already 80% done! There is token verification using the x-csrftoken and sessionid cookies (which I’ve talked about earlier). Simply apply that verification to market endpoints, add a limit of requests for each user to lets say 20 per minute and create API tokens for services you trust and want to see grow (for example earth2stats.xyz).
Writing this article took a lot of my precious time… If my goal was to be destructive, I’d just stay silent and try to hack the hell out of them… I believe in the potential of this project! The marketing team is doing a really good job, I just want to help push the dev team to do the same. I know they could... Thanks for reading and please comment below what you think!
"Destroy All Humans" by brunkfordbraun is licensed with CC BY-SA 2.0.