Source code distribution

edited October 2014 in Modding and Coding
How is the source code going to be distributed? Will it just be a straight source dump with the binary releases?

If that were the case, then personally I would commit all releases as updates to the trunk branch of a git repository. I would branch off from that trunk to make any changes I want, and integrate back changes from later releases. I think it would be a benefit to the modding effort if a DCVS were embraced in some kind of official way, to create a situation where it's a little easier to exchange and merge patches with each other.

Depend on Gavan's preferences, this wouldn't need to be the actual development repository, with all of the day-to-day changes, but just an official releases repository with a single branch and commits corresponding to every update.

One way to implement this might be as a private GitHub repository, allowing anyone with a license key to be granted access on request. (Although in principle the *only* distribution option probably shouldn't require signing up for GitHub, so source dumps could always be provided, maybe.)

edit: The answer (posted October 26):
gavanw said:

I'm putting it in a public github repository. Pirates will a way either way, so I'm not concerning myself with DRM or protecting the code. However, if somebody attempts to use the code then resell it without licensing it or requiring VQ keys, and their product becomes at all popular, I will probably send a cease and desist (again, I hate legal stuff but at this point you can't give others total free reign, or else they could put me out of business by undercutting my prices, etc). Also as mentioned: once I get enough money to keep the company afloat indefinitely, I would like to go full open source under a permissive license (ZLib, MIT, etc).


Comments

  • Just to understand - are you trying to solve the problem that mods need to be updated as the game is updated? So if we have, say, full access to the git repo of the game, then applying commits from that repo to your mod (which is maybe a branch in your copy of the git repo) is easier?

    If yes, that makes sense to me.
  • I kind of like that idea... it makes sense, yes. It's a lot better than some of the alternatives.
  • tyler said:

    Just to understand - are you trying to solve the problem that mods need to be updated as the game is updated? So if we have, say, full access to the git repo of the game, then applying commits from that repo to your mod (which is maybe a branch in your copy of the git repo) is easier?

    Yeah, that's what I meant. It seems like it could simplify things a bit.
  • Actually, if you're expecting to need the changes merged into your mod, isn't that a bit too much coupled with the engine ? I mean, mods should probably use a stable interface to the game, but not have direct access to its innards. Otherwise, I don't see how you'd manage the mess that mod compatibility would become.

    Just look at Skyrim and how everyone needs to thread so carefully in order not to disturb this or that mod.
  • I understand your concern, but the thing is, as we have the source code, it makes sense to expect people to change bits of the source code along with their mod. Bethesda never released Skyrim's source code or we'd probably be seeing source code mods there too.
  • With great power comes great responsibitity ! :P And direct access to the sources does gives a lot of power to modders.

    If we do wish for some mod intercompatibility down the road, something should be done about this. Unfortunately, I really don't have any idea at this point how this could be done. If someone really needs to manually merge all the mods manually to build the combinations she wants, I don't think people will try to integrate any other mods except the most popular ones.

    Maybe we could use a concept of integration levels for mods. For example, one mod that is heavily coupled with the sources would have to live on its own but would probably still support higher level mods, or something like that.

    Another idea would be to have some code logic that gives the opportunity to make changes to the behavior of the engine, but without changing the code. Really, the best analogy I can think of right now would be the Scala traits or Ruby mixins (AFAIK). You can change the behavior of current implementation, but without touching their code.

    Since @gavanw‌ specified that the sources should be available soon, I guess some thoughts should be put on this problem sooner rather than later.
  • edited October 2014
    I'm putting it in a public github repository. Pirates will find a way either way, so I'm not concerning myself with DRM or protecting the code. However, if somebody attempts to use the code then resell it without licensing it or requiring VQ keys, and their product becomes at all popular, I will probably send a cease and desist (again, I hate legal stuff but at this point you can't give others total free reign, or else they could put me out of business by undercutting my prices, etc). Also as mentioned: once I get enough money to keep the company afloat indefinitely, I would like to go full open source under a permissive license (ZLib, MIT, etc).
  • gavanw said:

    I'm putting it in a public github repository. Pirates will a way either way, so I'm not concerning myself with DRM or protecting the code. However, if somebody attempts to use the code then resell it without licensing it or requiring VQ keys, and their product becomes at all popular, I will probably send a cease and desist (again, I hate legal stuff but at this point you can't give others total free reign, or else they could put me out of business by undercutting my prices, etc). Also as mentioned: once I get enough money to keep the company afloat indefinitely, I would like to go full open source under a permissive license (ZLib, MIT, etc).

    I really hope it gets to this point. With an open source community, we'd be pretty sure to see a bunch of voxel games pop up everywhere.

    I tell you, the voxels are the future ! (Uh, you knew that, how odd :P)
  • Compatibility is a good point to mention, ASalvail. I think there will eventually wind up being three different "tiers" of mods:

    1. The higher-level tier: These change parts of the source code, sometimes resulting in you playing an entirely different game. Minimum compatibility with other high-level mods, due to the fact that they change the same code. Compatibility isn't impossible, but it wouldn't always work.
    2. The mid-level tier: These add auxiliary bits of code like a regular Skyrim mod and access higher-level engine functions. They'll be compatible with some high-level mods, but incompatible with others; it will depend entirely on which bits of source code they run on. In the same way, they'll be much more compatible with each other, and whether or not they'll work with another mid-level mod will depend mostly on whether that mod works with the same high-level mod.
    3. The low-level tier: These do little more than add simple lists or assets to the game. As a result, any low-level mod will work with almost any mid-level or even high-level mod, but they won't change or add much to the game as compared to the other tiers. A good comparison would be a Dwarf Fortress mod that adds new creatures or stone types.
  • This is probably the most feasible way to do it, Talvieno.
    How much should the game support/enforce these tiers ? Would we want to make a suite of test your mod needs to pass in order to be considered for a tier ? It would help compatibility, but I wonder how much energy it's really worth spending on that matter.
  • No, see, that's the beautiful thing. It'll happen by itself. Any mod that ever gets made will fall into one of those three categories: Changes, additions, and additional content. As a further example, some mods will have changes and additional content - so that gets classified as a high-level mod, just because the changes are a higher level.

    Modders test for intercompatability as a rule anyway, so we don't necessarily have to worry about it ourselves. It's in a modder's best interests to make their mod compatible with as many other mods as possible, so you can assume it will be done.
  • I'm a little new to this discussion, but I was actually thinking about this earlier today and here's what I came up with:

    Gavan writes the vanilla version. Takes a lot of time, and it would probably be a pain in the butt for him to rearrange the code to make it easy for all types of modding. So someone else releases a rearrangement of the code (the "mod manager") that allows toggling certain sections of the source code on and off (the AI bit, for example, or the story, character stats, or whatever). Mods can fill in for those sections with their code, and the "mod manager" knows which mod tries to change what (and what its changes are linked to). If two mods try to change the same part of the source code, the player has to pick which function to use. This is for the high-level tier. If these conflict, player resolves conflict before game starts. If they don't, no problem.

    As for the mid level tier, I think it could work the same way? I would define a mid-level mod as a mod that doesn't take away from core functionality provided by the engine, but adds something extra (a new context menu or AI processing step). Each mod specifies where it would like functionality to be added, and it's unlikely that those would conflict. If they do, player chooses how the engine integrates mods.

    The lowest tier is simple. Let each mod specify a list of new assets to augment or replace core engine assets, and there should be no conflict.

    Let me know what you guys think! I'm sure the idea could be refined a bit, but I would definitely be willing to write some kind of mod manager to do the job. This is just how I think it *could* work.
  • That's about what I meant, but the mod manager is a brilliant addition. We can only hope someone comes along and makes it - and you get a +1 in my book for stepping up and offering. :D Gavan's work should probably be focused elsewhere, especially if he's releasing the source code.
  • I think that it's the kind of thing to establish a "standard" for early, you know? And of course not everyone would choose to use it, but it would hopefully make compatibility and even mod development easier to handle. As the source code comes out and people start messing with it, I'll look at the directions that people take and see how viable a manager would be.
  • If VQ already includes (a basis for) an API, couldn't modders contribute their own code to this API, which they're using in their own mods?
    This would get you peer review of any code added, and the modder is sure the hooks and methods he needs are included in the base game.

    Above this layer you'd indeed need a mod manager through which to turn on or off mods or features of mods.

    I think it would be in the best interest of modders to categorize their own mod submissions. If you're changing worldgen, you need to let people know, so they can avoid other mods that change the same worldgen. Perhaps the API can help with this by alerting the player that a mod already hooked into the landmass generation and that the 2nd mod trying the same hook could lead to conflicts.
  • edited October 2014
    How do you imagine the API being shared among modders though? If you have a separate repository for the API, it seems to me that it would become rather big (if lots of people contribute mods) and each mod requires its own special hooks and methods anyway. Unless there's some sort of central standardized release of the API, I'm not sure how modders could add their own code. That said, I definitely support an extended API geared towards modding!
  • From what I understand the VQ source would be hosted on some kind of github solution. I'm not sure if an official decision in this regard has been made though.
  • Katorone said:

    From what I understand the VQ source would be hosted on some kind of github solution. I'm not sure if an official decision in this regard has been made though.

    Yes, it will be on github - to tell the truth I am far from a pro with github, have only used it for very basic tasks. But like most source control, you can create branches of the original project (in github terms, I guess this is called "cloning"). Given proper permission, you can also merge changes back into the source branch (or push them out for approval). Again, this is just from my modest understanding of github, let me know if I am incorrect.

    There is no API to isolate behavior, because in my experience APIs just get in the way - you end up needing to make XYZ accessible, and it is a whole lot of redundant code.

    My plan is to intelligently isolate chunks of the program with classes, and changes to these classes should be pretty straight forward.

    I tend to avoid unnecessary classes, polymorphism, and extending classes. Instead I prefer uberclasses (classes that cover all scenarios of whatever they are trying to abstract) -- different from uberclassing I mentioned in the RPG context, of course. :)

    A lot of modding will be doable not touching the C++ code, but just the json files that store definitions for things in the world. Basically editing the property sheet that specifies all the rules, parameters for world generation, item definitions, etc).

    Additionally, a lot of modding can be performed just messing with the GLSL.

    Early on, changes are going to break things, it is somewhat inevitable. But as long as you stick with the same source branch you will of course be fine.

    There also needs to be a trusted site for distributing/validating mods. Since full source goes out, people could create viruses or malware. We will need to create a site to distribute mods from trusted parties (of course, people can determine who they trust on their own as well and download the mod from a site they trust).
  • edited December 2014
    Depending on the quality of the engine's source code when it is released, I may consider funding gavan for more development or licensing the engine for development (not sure if that was included in the kickstarter as a one-time fee or what). It depends on quality that is either there or not, and what I can get out of it. A few issues may be I may need to re-write some of the heavier internals or not depending on seeing what I can improve or not in particular using agents to distribute the voxel creation and I will have to look at a few other procedural voxel engines.

    But I am definitely more towards improving/working with gavan rather than working against him.
    There also needs to be a trusted site for distributing/validating mods. Since full source goes out, people could create viruses or malware. We will need to create a site to distribute mods from trusted parties (of course, people can determine who they trust on their own as well and download the mod from a site they trust).
    The way I saw distribution being accompanied better is if we work on an internal scripting language for the modding game that is "safe" for modding but allows decent freedom. I've actually read some research papers on fully isolated modding programming languages and I think I know how to construct something decent.
    There is no API to isolate behavior, because in my experience APIs just get in the way - you end up needing to make XYZ accessible, and it is a whole lot of redundant code.
    API in my opinion would be the scripting code embedded within the engine if developed probably after other complexities the way I see it. So it'd be very powerful and yet it would be completely seperatable from the engine. It would also not take up more space as it'd allow the coder to do more with less code like Lobster programming language on github which can code minecraft in >1k lines of JS I could see a much more polished version of this game engine do something similar.

    And people will always try to cheat the system regardless though, so although we can set prices on addons etc they will most likely be pirated eventually which is just the nature of the markets these days. There is really not too much that can be done to prevent the pirating, but drawing up terms for pirated assets might be possible in a software license so it may make sense to read unity or unreal's licenses for those things.

    I definitely plan on putting more effort into helping his project rather than profiting from it, but I plan to do a bunch of work on the engine if he allows it.

    I know it may be too early to discuss a lot of these things, but I think me being open on what I want out of the project and helping putting the work into it I think we can get better results. I wanted to get this out here so he knows where I stand on the whole ordeal and we will discuss terms or how to work things out as development continues.
  • @Switch33‌ If I go with any scripting language, it will probably be Lua, just because it has been hardened by lots of real-world use, and is a lightweight solution for scripting in games. I have very little Lua experience (in comparison to experience with things like JS). Still, the old method of passing JSON from a web browser is still there, so you can pass JSON in realtime to the client over a websocket and back, as with the material editor. See:



    This is not very useful as a scripting language, but the functionality is there for people to do interesting things with (like hooking it up to a pizza delivery API).

    As for licensing, I consider any deal that comes my way (and a few have already, although I have not yet accepted any outside deal to date). The default licensing agreement is that you and your users only need a Voxel Quest game key, and from there you can do whatever you want with the engine (a bit similar to what Valve does with the Source Engine). But I also consider royalty deals, flat rates, and subscriptions among other things. As I have stated elsewhere, my primary goal is to raise enough money to sustain the company indefinitely, at which point I would like to open source the engine under something permissive like MIT, Zlib, CC, etc. If/when it goes under such a license, I would still like to find ways to fund development (the app/asset store was one idea), another would be selling superficial/vanity items like Team Fortress does with hats. Open to other ideas as well. I think the biggest potential is opening this thing up to the community, something Mojang should have done IMHO (and maybe Microsoft plans to? They've been on an open-source streak lately). Raising money is a secondary goal for me, but nonetheless important for growing this game, the company, etc.
  • edited December 2014
    If I work on the networking i'd try to do it through erlang. It seems to have the biggest rate of scalability, and I've found decent enough libraries for it. As for alternative web based backend i'd use Lua on top of JS through tessel: http://blog.technical.io/post/102971943072/luajit-on-tessel-first-steps . The only other viable alternative to lua on top of JS for a non-native runtime(something that primarily relies ontop of JS) would most likely be DartVM because it will be a bit more lightweight and eventually support multiple threads for JS, however the support for DartVM in my opinion is trailing that of lua.

    That being said, lua is more tested and can do pretty decent for networking as well. It used to be the go-to language for networking any game on the web for a long time. So work on things and maybe I can just make an add-on or wrapper version for the networking later. I don't mean to be a pest on such a trivial issue (in the sense that it will probably scale fine with lua).

    As for a database for the main server I think something fast, yet cheaply scalable might be aerospike: https://github.com/aerospike that recently came out. It looks pretty cool as it's the first distributed cache NoSQL database with SSD optimization . And then you can find a cheap SSD hosted provider online (like this one: https://news.ycombinator.com/item?id=8702966 ). You can probably worry about using a different database later though, not as important for first few releases. You can also definitely get away with a several non-SSD postgres or couchdb databases hosted on amazon for when speed isn't as important. I think aerospike will end up being a mad cheap for a database hosting service with great speed is only reason I am mentioning it.

    As for the licensing, definitely gonna be a more approachable topic when the source comes out. I just thought it'd be humbling to know I am a bit interested. I am definitely for sharing resources/links etc. I'm always overloaded with finding the best sources of information etc because it supports me as an investor at the moment.

    Microsoft has been making some neat things with C# like parallel Linq. I agree in that they are somewhat on an opensource streak lately. I wonder if they will opensource minecraft eventually with some newer features.
  • edited December 2014
    @Switch33‌ consider me humbled :) I also read over your intro thread - I used Warcraft 3's modding tools as well and really liked them but never built anything with it. Anyhow, you seem to know a lot about many aspects of programming/AI/etc (much more than I did at 23, and quite a bit more in many areas than I know about now). :) I did not really get that deep into game programming until I was about 23 (even though I started programming at around 13) - I am now 33.
  • edited January 2015
    I also started programming at a young age around 12-13 or so.

    Most big label game companies have chosen to use sever-client as there is less to deal with when servers get attacked because in p2p everyone gets attacked instead of servers.

    A good overview on service architectures: https://www.oasis-open.org/committees/download.php/19679/soa-rm-cs.pdf

    Lua final version (Though I think most of the methods can be carried over using erlang even done better. I am writing both versions of these docs to draw comparison and look at methods.):


    Queuing and client/server implementation:
    Distributed server architectures often use queue implementations. The reason to use the queue based method is it will allow all loading to be either synchronous or asynchronous, and it will also allow future scaling of complex client-server deployments.

    I think a good server-client architecture that would suite well with scaling would use queues. An example that uses ZeroMQ: https://github.com/TTimo/es_core for the server and the client interface as simple as: https://github.com/zaphoyd/websocketpp , which you can read about here: http://ttimo.typepad.com/blog/2013/05/es_core-an-experimental-framework-for-low-latency-high-fps-multiplayer-games.html

    It can also work like joyent's minecrab project and eventually suport real-time save/load massive environments: https://github.com/joyent/minecrab . This can probably be done by using same operating systems that joyent uses in a cloud environment which is illumos and optionally some of their other repos. illumos is built from solaris and is often a good choice for a cloud deployment for scaling with lots of IO. (There will be not much of any extra work required to do this as well.)

    What happens when the server queue fails?
    With a queue you can have a failover method for spilling files out on the server on-disk for later re-processing. Blizzard has an example repo of this here: https://github.com/Blizzard/FailoverQueue

    Tarantool spills over using the queue implementations it has. There are three of these queue implementations: https://github.com/tarantool/queue , https://github.com/tarantool/queue-collection , https://github.com/mailru/tntlua . Of these tntlua is the critical one that is on-disk and eats up memory the most.

    Netflix's hystrix is also a good example to follow about limiting access to certain things so dependencies don't overload servers with requests: http://www.infoq.com/news/2012/12/netflix-hystrix-fault-tolerance , https://github.com/Netflix/Hystrix/wiki/How-it-Works#Isolation
    More specifically, there needs to be a circuit breaker pattern implemented: https://github.com/yammer/circuitbox . For tarantool it uses some as queue managers: https://github.com/tarantool/queue

    How to start building this way?:
    A good start for the client with a performant caching database using nodejs, lua, and redis: http://tarantool.org/

    (some documentation may be in russian, which you can translate somewhat accurately in google translate)

    It should have built in support for anyevent and AMQP support (this is basically done for you): http://onperl.ru/onperl/temp/hlpp-anyevent-2.pdf , http://2012.nosql-matters.org/cgn/wp-content/uploads/2012/06/tarantool-nosql-matters-2012.pdf

    Then eventually build based off ideas under "loading things asynchronously using tarantool:"


    Preventing Cheating/some malicious activity:

    The tracerbullet architecture is from the book: Ship it! A Practical Guide to Successful Software Projects . It is worth skimming over if you can find a version of it online for just the single chapter it has on it.

    If your like me you would rather just look at real software examples as the books usually trail behind a bit though. So I found this blog post: http://atbrox.com/2012/01/24/tracerbullet-hadoop/ It is a good overview of the older ways of setting up authentication over the servers and what to seperate. In reality what it means is there should be seperable processes and loggers that can communicate if something fails, but also hide the real server infrastructure (dns records).

    Tracing things I think it is possible to use some of zipkin's code. Tracing can be a bit complex with microservices (the only real downside, but not major). Tracing code for redis in zipkin is here: https://github.com/twitter/zipkin/tree/master/zipkin-redis

    Loading things asynchronously using tarantool:

    Besides most modern web development needs to have a choice between asynchronous and synchronous, there should be good IO performance which usually means using Libev and nodejs as they are the most heavily optimized with support at the moment. I think libev beats out libuv currently.

    Managing DNS using async libraries libev and libcares: https://github.com/Mons/libevares , https://github.com/Mons/libevconnection

    Managing tarantool for specific parts using async libev: https://github.com/Mons/EV-Tarantool

    Tarantool can do persistent queues in order to perserve the rendered world: https://github.com/tarantool/queue-collection

    How to do queries on tarantool asynchronously?:
    Queries in microservices generally follow the CQRS (http://msdn.microsoft.com/en-us/library/dn568103.aspx , https://github.com/mspnp/cqrs-journey-code) which allows you to read/write at the same time without issues. For tarantool queries are seperated by queues. But you may want to use CQRS as an alternative: https://github.com/fanchy/fflib/tree/master/fflib/cqrs

    For complex procedures using the database you can use stored procedures(in memory queues): https://github.com/mailru/tntlua , https://en.wikipedia.org/wiki/Stored_procedure , http://2013.nosql-matters.org/cgn/wp-content/uploads/2013/05/kostja_tarantool_2013_cologne.pdf


    How to structure background jobs (and eventually rendering):
    As for storing voxel data in memory or as a file format I think this is a good example: https://github.com/imageworks/Field3D
    I found this which is a good breakdown of how to do rendering in specific parts, though it'd be using lua instead of js: https://github.com/naturalatlas/tilestrata
    Staging/using the voxel render: https://github.com/tsubo164/Fujiyama-Renderer , http://www.smallvcm.com/ , http://www.openvdb.org/ (vertex rendering and combining in real-time)
    How to structure very large terrains: https://tel.archives-ouvertes.fr/tel-00461667/file/091217_thesis.ppt , http://research.microsoft.com/en-us/um/people/hoppe/geomclipmap.pdf , https://code.google.com/p/gpu-geometry-clipmaps-minimal-terrain-engine/

    What deserves to be built in a background job? Anything that's not always super fast:
    Warming caches or Counting disk usage
    Building tarballs and sending tarballs
    Firing off web hooks (could be used for the game asset store)
    Creating events in the db and pre-caching them
    Building graphs
    Deleting users
    Store geo-location data and decide on which tarantool node sizes/players
  • edited January 2015
    Scalable build systems:
    Build systems for deployed items should be able to resolve dependencies and they should not require complex makefiles or configs:
    A good build system for lua: https://github.com/gittup/tup
    A good build system for cplusplus: https://github.com/andlabs/qo (though it is still in the works a bit)


    subversioning specific assets using operational transforms:
    You can avoid JSON conflicts using operational transforms: https://github.com/Operational-Transformation
    A good example of using graphs with json only as well: https://github.com/avoidwork/tiny-graph
    JSON arrays; objects formats: https://github.com/mashery/iodocs#example-2---request-bodies-arrays--objects

    Geo replication:
    One of the goals of managing a distributed service architecture is managing zones to improve latency and load balance.
    Netflix for example uses ribbon (http://techblog.netflix.com/2013/01/announcing-ribbon-tying-netflix-mid.html ) , but tarantool has it somewhat built into the service.

    Delegating authentication for before-connecting to services can be done through a radius server implementation using tarantool: https://github.com/zloidemon/tarantool-radius

    This can also be achieved by using Amazon cloud services and choosing several zones for back-up nodes.

    Apparently tarantool can even do node pooling and database sharding which means the databases can be distributed between all the clients and can be delegated using authentication.

    Conclusion:

    Eventually tarantool could even be re-written to use a more scalable versioned b-tree: http://vldb.org/pvldb/vol5/p884_benjaminsowell_vldb2012.pdf
    This would allow save/loading and concurrent world editing with subversioning of the worlds. Similar to what this group tried to do but could only scale to a few players editing the world at once: http://the-witness.net/news/2011/12/engine-tech-concurrent-world-editing/

    Future Design/Updates:

    Just announced not too long ago: https://gigaom.com/2014/12/15/microsoft-open-sources-cloud-framework-that-powers-halo/
    (It is probably not as advanced as the lua or erlang models though as .NET is usually trailing the efforts in those 2 languages)

    It may be something to help improve implementation details a bit depending on if it is any good or not(each player involved in gameplay being connected in a distributed actor framework that would spread the workload). I think what I recommended before in the post above is a very solid start though that will work regardless of new software tech design that comes out.

    Actors can be more or less important depending on the role usage decided for them. While I think it is possible to have the whole world procedurally generated as each player being counted as an actor, that would be a bit overkill and I do not recommend going down that path at least this early on.

  • edited January 2015
    ---start of what will be eventually the final overview/structure---

    Networking with Erlang

    Alternatively an Erlang version is already somewhat built/constructed using riak. This would probably be better for scaling, and with some work can provide the same level of atomic updates. Erlang also comes with good solutions for both chat and physics that are already developed.

    Erlang Books worth browsing:
    http://littleriakbook.com/
    http://www.erlang-in-anger.com/

    Basics:
    How to build erlang:
    Erlang has good build tools that are not overcomplicated and resolve dependencies well: https://github.com/rebar/rebar3

    Debugging Erlang:
    https://github.com/ferd/erl_crashdump_analyzer
    https://github.com/ferd/recon
    https://github.com/lostcolony/damocles

    Slow intra-nodes, drivers, and speeding up erlang:
    Despite some things that may seem as a disadavantage there is still a lot that erlang offers. Most real-time game development is moving to erlang.

    Erlang is more built for distributed systems than other languages. Erlang however has "slow" inter-node communication. This can be fixed by writing drivers to do NIF(network interfacing) done right. Optimizing code for native erlang is a good overview of this: https://www.youtube.com/watch?v=57AkoJfojK8&feature=youtu.be

    Some examples of these drivers are erlport: http://erlport.org/ , but something that can be used as-is with C++ is tinch: http://www.adamtornhill.com/code/tinchpp.htm

    An example using boost for intra-node communication for erlang with benchmark: https://github.com/davide/erlang_mq_boost

    HTTP/Web implementation:
    How to deal with the slowness that is HTTP when it is used?
    HTTP is actually relatively fast with some implementational adjustments. There are in general 3 choices: http://dsheiko.com/weblog/websockets-vs-sse-vs-long-polling/ Of which generally websockets are what you want but long polling/short polling are fallbacks.

    From the development of websockets there is now a more managable solutions to a safe performant http connection manager ( https://stackoverflow.com/questions/19169427/how-websockets-can-be-faster-than-a-simple-http-request ).

    The websockets are cached using an erlang falvor of redis: https://github.com/imvu/dtm-redis This version of redis using erlang and has support for transactional updates which can allow for shared node pooling updates. Logging in and saving some things like account information and skills/items should be done through this with logging.

    Erlang stack for websockets that would be used:

    Websockets as well as node to node programming for erlang have cap limits, but they can managed.
    webmachine: https://github.com/basho/webmachine
    To fix overloading webmachine your app talks to webmachine which in turn talks to riak: http://buffered.io/posts/webmachine-erlydtl-and-riak-part-3/ , https://github.com/spilgames/spapi-router , handling ips: https://github.com/mochi/egeoip
    Webmachine serves as the framework for maintaining connections to riak as well as the framework for connections to websockets:
    A comparison between webmachine and N2O is here: http://lists.therestfulway.com/pipermail/webmachine_lists.therestfulway.com/2010-December/000428.html
    So you may want to look over some of N2O's resource handling as well, but webmachine is a better fit for managing connections.

    Caching the socket connections and using pub/sub architectures for management:
    Redis The websockets are cached using an erlang flavor of redis: https://github.com/imvu/dtm-redis This version of redis using erlang has support for transactional updates which can allow for shared node pooling updates.
    (If this thing is built right, you would probably need only need roughly 2 redis clients to handle over 260,000 + connections based on how stackoverflow runs!)

    Accepting connections:
    https://github.com/ninenines/cowboy (http server)
    https://github.com/ninenines/ranch (tcp acceptor)
    https://github.com/extend/bullet (streaming)
    https://github.com/extend/gun (Gun is the asynchronous client)
    https://github.com/sockjs/websocket-multiplex (Making a single access point for several websockets)

    Rendering webcontent from within the game engine can be done using an fast cgi client:
    https://github.com/extend/ex_fcgi

    You also may want to use librocket: https://github.com/libRocket/libRocket

    Some notes about websockets security: https://gist.github.com/subudeepak/9897212

    Limits of Scalability, and workarounds:

    With the websockets implementation using HAproxy a lot of the security is done for you. HAproxy has a cap on websockets: http://serverfault.com/questions/642256/scale-haproxy-for-more-than-64k-websockets
    Haproxy: https://github.com/haproxy/haproxy
    Basically you need to route them using haproxy which is detailed mid-way through this blog post: http://buffered.io/posts/webmachine-erlydtl-and-riak-part-2/

    Poolboy (a better version of pooler) can also handle multiple connected database pools to riak. Haproxy's limit can then be avoided: http://buffered.io/posts/webmachine-erlydtl-and-riak-part-4/ , https://github.com/devinus/poolboy ,https://github.com/puzza007/riakc_poolboy

    Websockets often fail at compression:
    Websockets fail at compression which you can read about here: http://moduscreate.com/fast-polling-vs-websockets-2/
    This is work-aroundable if you use polling instead. Tarantool does long polling here: https://github.com/dr-co/lp

    A free specialized protocol for real-time chat rivaling that of skype:
    ejabberd is currently the leader in scaling chat: https://github.com/processone

    Readymade physics distributed:
    Erlang bullet physics: https://github.com/nialscorva/erlybullet

    Managing multiple backend databases:
    Managing riak can be used with a statebox for managing conflicts/resolutions using statebox.

    Riak handling statebox for managing multiple backends: http://basho.com/statebox-an-eventually-consistent-data-model-for-erlang-and-riak/, https://github.com/mochi/statebox , https://github.com/mochi/statebox_riak , https://github.com/mochi/recordset
    Requirements: https://stackoverflow.com/questions/16122480/how-to-combine-statebox-and-map-reduce-in-riak

    Queues and process groups with riak:
    Riak:https://github.com/basho/riak
    Riak's keyvalue storage: https://github.com/basho/riak_kv
    Rabbitmq is the most performant of messaging queues.
    A rebar friendly client for AMQP: https://github.com/jbrisbin/amqp_client
    Riak with a persistent rabbitmq implementation: https://github.com/jbrisbin/rabbit_riak_queue

    Sticking messages in riak is also relatively simple: https://github.com/jbrisbin/riak-exchange
  • edited January 2015
    Replicating data types: https://github.com/basho/riak_dt

    Data retrieval:
    Asynchronous querying for the database:
    Querying from a database asynchronously would use the same CQRS idea as the lua version: http://www.infoq.com/presentations/cqrs-erlang

    Structuring JSON:
    Avoid JSON conflicts using operational transforms: https://github.com/Operational-Transformation
    Using graphs with JSON: https://github.com/avoidwork/tiny-graph
    JSON arrays; objects formats: https://github.com/mashery/iodocs#example-2---request-bodies-arrays--objects

    JSON and blobs:
    (I actually forgot to mention eblobs before in the lua version, but I will draw a comparison here. They make sense for when JSON can be too slow for transfering large objects). eblobs is having large binary objects transfered instead of json.

    An example of when to use blobs vs json for sqlite is a good overview for size for when something is considered a blob or not: https://www.sqlite.org/intern-v-extern-blob.html
    Most eblobs (binary objects) have good reference implementations here: https://github.com/assimp/assimp , json backend(use eblobs for large files) https://github.com/acgessler/assimp2json, loading textures: https://github.com/acgessler/half_float

    There are somethings that are more attuned to how erlang handles eblobs below this text:
    Storing erlang beams in memory for reading similar to tarantool's in memory queues containing unread data is too slow!: https://github.com/mochi/beambag,so instead to load large files into memory using a driver based method that basically just uses a reference skipping some of the extra erlang baggage of reading and writing the memory for the files would be cherly: https://github.com/openx/cherly

    Caching:
    Caching is important for rendering large amounts of data in-memory. There are many choices in this retrospect.
    A comparison of key value database performance: http://influxdb.com/blog/2014/06/20/leveldb_vs_rocksdb_vs_hyperleveldb_vs_lmdb_performance.html

    Of these Rocksdb seems the most performant and what should be used for a cache solution:
    Using NIF interface for erlang:
    https://github.com/ttyerlsol/rockse (may require a bit of work)

    other cache structures using NIF:
    erlang NIF data cache structures: https://github.com/davisp/johnny (these are for in-memory database structures similar to what tarantool does, good reference)

    Compression:
    Using NIF with erlang and compression: https://github.com/ttvd/wflz-erlang-nif (useful basic compression)
    IMVU also has a good library on Encoding vectors: https://github.com/imvu/normal_encoding , http://faculty.cs.tamu.edu/schaefer/research/normalCompression.pdf
    Better compression for random values: https://github.com/Cyan4973/FiniteStateEntropy
    Erlang image processing:https://github.com/mochi/erl_img

    Minimal Configs:
    lua can sort of be used to a certain extent (like config files) with a implementation of lua on erlang: https://github.com/rvirding/luerl

    Other hackish attempts at making erlang smaller for configs:
    https://github.com/s1n4/eake
    https://github.com/maximk/teeterl

    Erlang fault tolerance and rate limiting:
    Sidejob provides a rate-limiting approach: https://github.com/basho/sidejob

    For sending real-time networking erlang provides good logging services: https://github.com/mochi/udplog Request tracing: https://github.com/puzza007/folgian

    Erlang's term storage can fail if overloaded but there is a workaround: http://erlang.org/doc/man/ets.html, https://github.com/dipthegeezer/ets_manager

    Redis can be traced using some code from zipkin: https://github.com/twitter/zipkin/tree/master/zipkin-redis

    Eventually a restart service more useful than resque specifically for riak: https://github.com/cmeiklejohn/riaque

    Internally authenticating wireless servers can be done using a simple radius server:
    https://github.com/ates/netspire-core

    Erlang has some more advanced distributed systems research and projects as well:
    Because erlang was constructed with the OTP system with distributed programming as it's main focus at first the language makes a lot of it relatively easy.

    One interesting project using erlang for non-sync requiring distributed systems is syncfree which will eventually be useful as a more advanced riak: https://github.com/SyncFree/antidote

    Something to note about erlang random:
    It is relatively broken, but an implementation using NIF for a binding for crypto would fair much better
    source: https://news.ycombinator.com/item?id=8837588

    Subsection: Erlang already built solutions as reference:
    This section can be ignored, but it is a good overview of some completed projects using erlang.

    Fully built servers with docs:https://jeena.net/t/GGS.pdf , https://www.erlang-solutions.com/upload/docs/21/mmog_in_erlang.pdf ,
    http://www.diva-portal.org/smash/get/diva2:447210/FULLTEXT01.pdf

    Erlang server repos: https://github.com/jordillonch/eggs ,
    https://github.com/SkewedAspect/precursors-server , https://bitbucket.org/skewedaspect

    Related:
    IMVU Scaling:
    IMVU has Cal3d forked: https://github.com/imvu/cal3d
    (cal3d is kind of heavy in that it requires STL, i do not know of any other animation library that is decent though)

    Things that need to be updated:
    Final rewrite is coming soon. . . (time till it is finished keeps changing as I am reading more things to improve and doing more research)

    I found a bit more things that will simplify some of the work required like an OpenGL middleware layer: https://github.com/Eyescale/Equalizer. The final doc will be mostly based off the erlang version the main sauce and then fallback to some already coded things. Near the end it may also include a subsection with some listed alternatives.

    Websockets section is basically finished with lots of corrections.

    Working on compression topic and adding in the rest of it.

    Will need to add section a more general section on geoclipmaps and layered rendering.

    Only the tarantool stuff that matters is there for the most part now.
Sign In or Register to comment.