GUN: Frugality with S3, Save 90% Your Costs

By: gunDB

10   0   1098

Uploaded on 08/12/2016

Interested in seeing a demo for your organization? Get in touch today, enterprise@gunDB.io . Or visit http://gunDB.io/ !

Comments (6):

By marknadal    2017-09-20

Wow! This is an incredible article. I do research and development for systems like this at GUN, and this article nails a lot of important pieces. Particularly there ability to jump to an old message quickly.

We built a prototype of a similar system that handled 100M+ messages a day for about $10, 2 minute screen cast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 . However, this was without FTS or Mentions tagging, so I want to explore some thoughts here:

1. The bucketing approach is what we did as well, it is quite effective. However, warning to outsiders, this only effective for append-only data (like chat apps, twitter, etc.) and not good for data that gets a lot of recurring updates.

2. The more indices you add, the more expensive it gets. If you are getting a 100M+ messages a day, and you then want to update the FTS index and mentions index (user messages' index, hashtag index, etc.) you'll be doing significantly more writes. And you'll notice that those writes are updates to an index - this is the gotcha and will increase your cost.

3. Our system by default backs up / replicates to S3, which is something they mention they want to perhaps do in the future. This has huge perks to it, including price reductions, fault tolerance, and less DevOps - which is something they (and you) should value!

There backend team is amazingly small. These guys and gals seem exceptionally talented and making smart decisions. I'm looking forward to the future post on FTS!

Original Thread

By marknadal    2017-09-20

Sorry the website turned you off :( what recommendation do you make for balancing both enterprise and developers? Currently we have two website, the other for developers: http://gun.js.org/

Better links:

(again) https://www.youtube.com/watch?v=x_WqBuEA7s8 100M+ msg a day for $10.

Doing distributed machine learning: http://myrighttocode.org/blog/artificial%20intelligence/part...

30M+ ops/sec performance testing: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...

1 minute demo of seamless multi-server failover during a live chat app: https://youtu.be/-FN_J3etdvY

1 minute demo of recovering from a complete server meltdown: https://youtu.be/-i-11T5ZI9o

You are right - trust should be proven by demonstration, not the blind faith of others.

We're also deploying out to customers with 1.5M users in production and customers with a product being shipped to 1.5K homes. And we're nearing our production-ready v1.0 stable release.

Feedback like yours is helpful, so please hit me up with any other critiques or ideas on how we can explain ourselves better. Thanks for jumping in on the convo :)

Original Thread

By marknadal    2017-09-20

Stuff like this is great! And you said you mentioned you are primarily doing it for timeseries data - then you can easily batch writes to handle high volume throughput too!

We did this with S3 as the storage engine, 100M+ records for $10/day: https://www.youtube.com/watch?v=x_WqBuEA7s8

And discord had a very nice article on this as well: https://blog.discordapp.com/how-discord-stores-billions-of-m...

Great work, I think there is a lot of exciting stuff you can add to it!

Original Thread

Submit Your Video

If you have some great dev videos to share, please fill out this form.