Dynamic linking is scary

One of my goals for the new version of Shogun was to create a language to go along with it where I used little to no reference material for the compiler. I’ve done pretty well for not having any background in creating such a compiler, if I do say so my self.

Originally, the design of Sholan (the Shogun Language compiler) made it such that imported scripts would all be compiled into a single binary; that is, the entire standard library would be included in every single program compiled by Sholan. This wasn’t exactly ideal, so I tackled the process of dynamic linking.

Specifying linkage

One of the very first issues I ran into was how to specify whether an import was an external dependency or not. The simplest way would be to simply add a keyword after the import statement which marked it as a dependency, but what about imports inside of the file you are importing? They might be separate libraries, or they might also be a part of the same import.

I solved this by adding “import modes”. There are five import modes: None, import, inherit, library, and export. When importing a new file, the file’s import mode is pushed onto a stack. When the file is not being imported anymore, then the stack is popped.

None + Import

The “None” mode and the “Import” mode, for all intents and purposes, are exactly the same. “None” simply means that the current import is the top-level file (the script originally passed to Sholan). When at one of these import modes, Sholan acts as if the code in the related file is a part of the initial file.

Inherit

The “Inherit” mode is a special mode, in that it doesn’t represent anything in and of itself. It is the default mode when importing any file. When Sholan’s kernel is asked for the current import mode, it simply skips this mode and reads the mode from the next one on the stack, thereby inheriting the next import mode.

Library

The “library” mode, as it name would suggest, is used to import a library. This tells Sholan to not compile functions inside the imported file, but instead to simply register the function symbols as usable.

Export

The “export” mode has a special function, and is generally only to be used in build scripts (see the next section). It marks all imported functions as being exports, which is later used to setup a library for building.

Building libraries

Due to the architecture of the import system, building a dynamic library requires a “build script” to be written, which is a simple Sholan program that imports all library files in the “export” mode, and tells the compiler to output the correct headers for a library. This is done as follows:

#build-exports
import "mylibrary.sl" export

This tells the compiler to build the export header, and then imports the library.

Linking the “wrong” way

As I said, I have no background in writing compilers, and along with that I have no knowledge of how dynamic linkage generally works. As such, I decided to tackle it in the simplest way possible. The virtual machine imports a dynamic library as normal memory, and then the program jumps to the beginning of that memory. The “export header” I talked about earlier is simply a set of operations at the beginning of each dynamic library that pushes the addresses of each function onto the stack, and then returns control to the main program. The program then stores the function addresses in memory and accesses them like it would normal variables.

The major downside to this approach is that it completely depends on the order that functions are defined inside libraries, and it also has no mechanism to prevent a library from being loaded multiple times. On the other hand, it works, and it is a good first step towards a true linkage system.

That’s all for now. Next time I might get into how the compiler works, or maybe I’ll just start some programming tutorials.

Playing with compilers

Over the past few weeks I’ve been working on a rewrite of Shogun. It is still a stack machine, but now includes two registers, a sane assembler, a mostly complete set of opcodes, and now even a language that compiles to it!

Shogun

Shogun has been rewritten in its entirety, though it does use snippets of the original here and there (specifically, Shogun::Object is almost cut-and-paste from Shogun::Data). Binaries still use the same header, but the version has been changed to 0.2.x. New versions of Shogun will refuse to run old binaries and old versions will refuse to run new binaries. This is necessary as first of all the opcodes have changed, but more importantly the entire layout of the binary has changed.

The single biggest change to both the VM and the binary layout is the way Shogun now handles memory. There are only two segments of memory: The stack and the heap. There is no separate program memory, as Shogun simply reads the program from the heap. What this means for the binaries is that they only have two sections: the header and raw memory. There is no separation between what an opcode is and what data in memory is. What is nice about this is that you can actually pre-store memory you need inside your binary, and it will immediately be available inside the VM as long as you know where to address.

Now, as for the registers I mentioned, there are only two. They each can only hold an unsigned integer (internally known as the address type). The first is known as PRI, or program index. It simply stores the next address for the VM to execute. The second is known as MMX, which stores a local position of memory. When a program is loaded, MMX is set to the first unallocated position of memory. The VM never uses MMX itself, so it is up to the programmer as to how to use it. Sholan uses it to denote the current scope’s local memory.

I’m probably not going to be updating the online Shogun sandbox, as I don’t want to put the horribly written memory limiters on the new version of SVM. In fact, I’ll probably be taking down the sandbox at some point simply because it is horribly outdated.

Some other miscellaneous improvements:

  • Labels and JUMPing are now sane
  • No more global variables; you can’t define variables in your assembly, you have to hardcode addresses (or use the stack)
    • This was mostly done as it adds a bit of complexity to the assembler and is useless for an actual compiler to use
  • New filename extensions: *.shasm for assembly and *.sx for binaries.
    • On that note, Sholan files are *.sl

Sholan

Sholan is a new project that was added to Shogun a couple of weeks ago. It is my experimental compiler for a language also known as Sholan. I’ve been designing this language around Shogun, and have decided to finally build a toolchain for it. The compiler is written in C#, much to my chagrin, as the lexer/parser generator ANTLR4 doesn’t currently have a C++ target. Luckily, Sholan should run fine on Mono.

As for using the language itself… I’ll be writing a whole lot more about it in some followup posts.

Moving on from here

Looking to the future, I’m mostly focusing on Sholan development. Building something that uses SVM rather than building the VM itself forces me to think about what I really need from it. Here’s a list of some of the things that are in the works for both SVM and Sholan:

  • [SVM] Dynamic library support – some way to tell AsmReader to import external dependencies
    • Currently Sholan just throws every imported file into one binary. Woohoo monolithic code!
    • I want to try to get this done without really adding much to the VM itself.
  • [Sholan] Not all operators are supported
    • Stuff like bitwise operations
  • [Sholan] Some sort of OOP system
    • Not sure how I want to do this, as it is a huge beast to tackle.

Now I’ll leave you with a small snippet of code from Sholan:

// FizzBuzz
import "lib/standard.sl"

entry {
	print("N?")
	var count = readline()
	
	for(var i = 1; i <= count; i = i + 1) {
		var output = ""
		
		if(i % 3 === 0) {
			output = "Fizz"
		}
		
		if(i % 5 === 0) {
			output = output .. "Buzz"
		}
		
		if(i % 3 !== 0 && i % 5 !== 0) {
			output = i
		}
		
		print(output)
	}
}

My relationship with Minecraft

Over the past few months, I’ve just seen everything to do with Minecraft slowly become shittier and shittier. First the whole EULA drama: I don’t care to take a side in this post, but at the very least it showed that Mojang seriously needs to reign in their developers. Specifically one who I won’t mention here, who in a public chat said some nasty things about server owners. Justified or not, employees should not be making public statements about their company.

Then came Mineorama. This one certainly wasn’t Mojang’s fault, but it is just one of a plague of similar events that has happened in the past few years and it certainly doesn’t help the community at all.

Now we have the Bukkit drama. First Mojang comes out as the secret owner of Bukkit, which is highly unethical. Developers were freely giving contributions to an open source project that was assumed to be under independent ownership. Then we have the DMCA takedowns due to a former Bukkit developer claiming that the license to his code was invalidated by the fact that Mojang allowed their closed source code to be distributed with his GPL’d code. The scary thing is, I actually sort of agree with what he did. I’d certainly feel pretty damn betrayed if I suddenly found out that a company secretly owned a project I’d dedicated a part of my life to. I’m not saying what he did is right, but I certainly don’t blame him.

All in all, Minecraft has not had a good year. Not even close. Now that I’ve made that point, I get to talk about me: I’ve had enough. Enough of the drama, enough of the bullshit from a company that has way too much money and doesn’t now how to manage itself. I’ve made quite a few friends, and I’ve had a lot of fun creating games that many people have played. At some point, however, I just have to say enough. I don’t want to be a part of this community any more. I’ve had my fun, but I have enough to deal with in my personal life without having to sift through a pile of shit drama every day.

This doesn’t mean I’m leaving Minecraft right away, in fact I will likely stay at my position at The Chunk for a few more months. I still have quite a bit of unfinished business to complete, and I’m not about to completely leave everything I’ve created behind, but consider this a notice of my future intent.

Thank you everyone, it has been quite a ride.

-Sam “Red” Bloomberg

Current projects

I haven’t posted here for a while, and it’s because I’ve been working on a lot of new stuff, most notably a brand new game. I’m not going to say exactly what it is, but what I will say is that it is a neat little 2D platformer built with Unity.

 

I’m also thinking about what the future for Dive engine is. It was mostly an experiment in the beginning, and it worked out pretty well, but there are numerous problems with it at the moment. I really want to get around to writing a successor to it, or even a 3D variant (adding 2D support afterwords). I’ve been thinking about the best way to do this, and originally I was going to go with C++ this time around, but I’ve gone back on that decision and I’m currently planning on using C# once again. For me, C# is just too nice of a language to pass up. I might go back on the decision once again, but at this point I really can’t say.

Graphdat

I could be horribly late to the party, but I was just shown this awesome service, graphdat:

Graphdat graphs

Graphdat is a service which allows you to easily make a dashboard for graphing different things on a server. Here, I have a dashboard for redxdev.com’s CPU usage, memory usage, disk IO, network IO, application info, nginx connections, and mysql connections.

Go check it out at http://www.graphdat.com/, it’s free for a single server!

DivePong

Since there aren’t really any samples on how to use Dive to actually make a game, I decided to start doing that myself. For the first sample, I have started working on a very simple pong clone. You can find the repository here.

Right now all that is implemented are the components for the player to control their paddle. You can test it out by setting up the project, launching the engine, and running the command “entity_create_template Pong.Player”. Use the up and down arrow keys to control the paddle. Not much, but it’s a start. I should have the sample done pretty soon.

Update: The game is almost fully implemented. It allows you to play a basic game of pong and it keeps score. There are a few bugs with the ball movement, but it is playable.

New Dive Engine feature: game assembly

This is something I’ve wanted to implement for a while but only just got around to doing, but Dive engine now separates engine logic from game logic.

As of the most recent commit (at the time of this writing), the engine code is in a separate project from the game code. The engine still compiles to an executable, but the game code now resides inside of a dll. This is a breaking change; anything on Dive engine previously will likely not work after updating to the latest version. It should be simple to port old code over, but it is just something to keep in mind.

In other news, I have removed the old tutorials as they are now very out of date, and it would be better to either just look through the code or to wait until I get around to writing new tutorials.