Managing configuration stuff with json – a complete solution?

In this article, I’m assuming you know what the json file format is (and, of course, have some knowledge of the C++ programming language).

Motivation

Most applications don’t run out of the box. They need to be configured, first. Think of a web server, for example. It often needs to know which port it’s supposed to listen on, or where it should store files it wants to cache. There are a few standard methods available for configuring an application:

  • Command line parameters, which are passed to the program by the operating system as arguments to main (and are usually called “argc” and “argv”, for “argument count” and “argument vector”). You would use those variables to temporarily change the program’s behavior, not permanently.
  • Configuration files, which, on Unix-like systems, are stored either globally inside the /etc directory, or locally, usually below the home directory (think $HOME/.application_name/config_file). Most applications feature both a global and a local configuration file.
  • System-specific configuration databases, one of the most prominent ones being the Windows registry.
  • Preprocessor flags which are specified when compiling the program. They’re usually permanent after compiling/installing the program and cannot be overridden.

In this article, I will describe how to use json to create applications which are configurable via command line, global configuration file and local configuration file. All at once, and with the same syntax.

sge’s json module

Before I can dig in to how json is used for configuration, I should explain how we represent json in C++. We will be using sge’s json module, which provides a json parser for files, streams and strings. It uses spirit to do the parsing and basically implements the context-free grammar structure which is depicted on the json home page. To understand the following, you should have the json homepage with its diagrams open, parallel to reading the article.

Simple types

The grammar on the json website tells you that a json value can be a string, a boolean, a number, an object, an array or “null“. Now, how do we map these json types to C++ types?

Everything except object, array and value itself can be mapped pretty easily. The json sge module defines the following types/typedefs (they’re all in the sge::parse namespace):

json type C++ type
int json::int_type (typedef for int)
int with fractional part json::float_type (typedef for float)
null An empty struct called json::null
string fcppt::string
boolean bool

The value type

Inheritance? …

As you can see, the “value” type should be able to hold exactly one of the mentioned json types (except another value). Take a moment and think about how you would define such a “value” type.

If you’re an object-oriented programmer you might think that you have solved the problem: use inheritance. Define value as an abstract base class and derive all the other types from it!

// Our base class
struct value
{
	// We have to define this so it's really a base class (C++ idiosyncrasy)
	virtual ~value() {}
};

// Derived from base class
struct int_type
:
	value
{
	int number;

	int_type(int _number)
	:
		number(_number)
	{
	}
};

struct array
:
	value
{
	std::vector<value*> values;
};

// ...

This approach works. But it’s not quite the right tool for the job. Let me quickly give you some reasons for this:

To be as comfortable as possible, we would like all the json C++ types to be copyable, so we can pass them around freely. Inheritance – at least in C++ – doesn’t really play well into this. You can already see this in the code above. The array class needs to hold values. But it can’t really hold them by value (no pun intended), meaning we cannot define the “values” member to be a std::vector<value> values; because that would cause slicing:

// Let's collect some values
std::vector<value> values;
// Add an integer (as defined above)
values.push_back(int_type(10));
// Let's see if we can output the integer we just inserted
std::cout << dynamic_cast<int_type &>(values.front()).number << "\n";

The code above will throw a bad_cast. Why? Because the vector holds objects of type value. Pushing a subclass into the vector will “cast” (slice!) it into the base class, losing information.

We could work around this using smart pointers and ptr_containers, but that would be really clumsy and we would need new and delete.

There is another, more idealistic, reason for not choosing inheritance to solve the problem. Inheritance is an option if you do not know how many derived classes of value there will be, and to allow the user to extend the functionality by deriving from our base class. This is not the case with json. It’s clearly defined how many “derived” classes we have, and the user shall not extend the functionality.

…no, variants!

So, what’s our solution? Well, if you’re a functional programmer, you immediately think of an algebraic data type (or variant) for our value type. Don’t be afraid, I do not require you to know what that is, I’ll explain it. You can think of a variant as a base class for a fixed number of derived types. A quick example should explain how you use it and what it can do:

// number holds exactly one of the three types mentioned
boost::variant<int,float,double> number;

// Now number is an "int" (and not a float or double)
number = 10;

// We can extract its value using the "get" function
std::cout << boost::get<int>(number) << "\n";

// This, however, will throw an exception, since we didn't store a double in
// the variant
// std::cout << boost::get<double>(number) << "\n";


number = 10.0;

// Now this works
std::cout << boost::get<double>(number) << "\n";

// We can put variants in a regular container!
std::vector<boost::variant<int,float,double> > numbers;
numbers.push_back(10);
numbers.push_back(20.f);
std::cout 
  << boost::get<int>(numbers[0]) 
  << ", " 
  << boost::get<float>(numbers[1]) 
  << "\n";

Variants are obviously copyable and they don’t use new/delete, so they are an ideal candidate for our value. With that in mind, we can complete our json-to-C++-mapping as follows:

json type C++ type
value
variant
<
	json::int_type,
	json::float_type,
	json::null,
	bool,
	json::object,
	json::array,
	fcppt::string
>
array
struct array 
{ 
	json::member_vector elements; 
};

where json::member_vector is a std::vector<json::value>

object
struct object 
{ 
	json::member_vector members; 
};

where json::member_vector is a std::vector<json::member> and json::member is

struct member 
{ 
	fcppt::string name; 
	json::value value; 
};

(yes, calling both the type and the member variable “value” is possible in C++)

Reading and writing a json tree

To read a json tree, you can call one of the following functions:

Function Description
json::object 
parse_file_exn(
	filesystem::path)
Parses the json file and throws an exception if it doesn’t exist (exn is for “existing”).
bool 
parse_file(
	filesystem::path,
	json::object &)
Parses the json file and returns true if the parsing succeeded. The result is stored in the object that is passed as second parameter.
template<typename It>
bool 
parse_range(
	It &begin,
	It end,
	json::object &)
Parses the iterator range delimited by [begin,end), returns true if parsing succeeded and sets the begin iterator to the place where the parsing ended (which might not equal end). The result is, again, stored in the object passed as third parameter.
bool 
parse_stream(
	fcppt::io::istream &,
	json::object &)
Parses the stream, returns true if parsing succeeded. The result is, again, stored in the object passed as second parameter. Note that fcppt::io::istream is a stream adapted to fcppt::char_type and can thus be used in conjunction with fcppt::string.

If you have an object and want to write it to a file, there is

fcppt::string
output_tabbed(
	object const &);

which uses tabs to indent the json structure in an eye-friendly way.

Manipulating the json tree

Let’s say you’ve read in the following json object (from a file, for example):

{
	"player-name" : "pimiddy",
	"camera" : 
	{
		"x-angle" : 16.0,
		"position" : [4.0,8.0,15.0]
	}
}

You want to extract the various values from the tree.

  • To extract the player name, you have to iterate through the object’s members and boost::get<fcppt::string> the value of the member “player-name”.
  • To extract the “x-angle” inside “camera”, you have to search the “camera” member, convert it to an object, then search the “x-angle” and convert it to float_type.
  • To extract the “position” and store it inside a std::vector<float>, for example, you have to go to “camera”, then search “position”, then convert the (heterogenous!) array “position” to your std::vector.

Clearly, this is repetitive and annoying. A helper function find_and_convert_member is needed! This function is used as follows:

// Read our file (just to show you how one of the parse_ functions is used)
json::object const &config_file = 
	json::parse_file_exn(
		FCPPT_TEXT("my_file.json"));

// Note: This is a double, not float_type. find_and_convert_member doesn't
// care, as long as it's a floating point type.
double camera_x_angle = 
	json::find_and_convert_member<double>(
		config_file,
		// Unix filesystem-like syntax here
		json::path(FCPPT_TEXT("camera")) / FCPPT_TEXT("x-angle"));

// Again, this could be a double or long double, too
std::vector<float> camera_position = 
	json::find_and_convert_member<std::vector<float> >(
		config_file,
		json::path(FCPPT_TEXT("camera")) / FCPPT_TEXT("position"));

That’s much more readable and more intuitive code! Note that instead of a dynamic std::vector, we could have used an array<float,3>, too.

json for configuration purposes

Command line

With what we’ve seen so far, we can load a json file into an
object and extract values from this object. So the greater
idea is to read in a json file and use the json tree as a database inside our
application.

But we want to take into account command line parameters, too. So, obviously,
need to modify the json tree that comes out of the parse_file
method. This is done with the following function call:

int 
main(
	int argc,
	char *argv[])
{
	json::object const &config_file = 
		json::merge_command_line_parameters(
			json::parse_file_exn(
				FCPPT_TEXT("my_file.json")),
			json::create_command_line_parameters(
				argc,
				argv));
}

There’s a little more to this than you might have expected. What’s this create_command_line_parameters? Aren’t the command line parameters already there, “created” by the operating system? Well, yes, but they’re of type char, not fcppt::char_type, and they’re stored in a clumsy char*[] array! So in the code above, we convert them to a std::vector<fcppt::string>, which is more C++y.

The function merge_command_line_parameters takes an existing json tree (in our case, read directly from a file) and “merges” the tree with directives contained in the command line parameters. The above code allows you to issue the following:

./my_program 'player-name="test"' 'camera/position=[16.0,23.0,42.0]' 'camera/x-angle=18.0'

The syntax should be very clear:

option-path=option-value

where option-path may contain a ‘/’ to indicate “descend into the object”. option-value has to be a valid json type, so it can even be a whole object!

./my_program 'camera={ "position" : [1.0,2.0,3.0], "x-angle" : 32.0 }'

There are two important things to keep in mind when using merge_command_line_parameters:

  1. Your system shell parses the command line first. And while doing that, it might respond to certain special characters. For example, ‘[‘ and ‘]’ have special meaning in the bash shell. Also, quoting stuff with “” gives unexpected results. So my advice is: Always surround your arguments with apostrophes.
  2. merge_command_line_parameters does some type checking when merging your parameters. This means, for example, that
    player-name=10.0

    will throw an exception, since the original type of player-name is string! It doesn’t, however, recursively check object types (yet), so the following is valid:

    camera='{ "lol" : "this works" }'

    since “camera” has type object before and after the merge.

User configuration files

We’ve seen that we can merge an existing tree with a single new value. The next step is to merge two json trees. This situation occurs when you read a global configuration file, but want to incorporate changes from a user configuration file. We can merge two trees with the merge_trees function:

json::object const &config_file = 
	json::merge_trees(
		json::parse_file_exn(
			global_path()/FCPPT_TEXT("my_file.json")),
		json::parse_file_exn(
			local_path()/FCPPT_TEXT("my_local_file.json")));

Now we’ve got a complete configuration system with user config files and command line manipulation, all in json! In the next section, I’ll explain how all of it fits together.

Writing back changes and putting it all together

One thing might bug you: What if I want to manipulate the json tree in my program and write back the changed values to my user configuration file? The json module provides two wrappers for this situation. One is the function modify_user_value:

void
modify_user_value(
	json::object const &global_configuration,
	json::object &user_configuration,
	json::path const &,
	json::value const &new_value);

It takes the global_configuration object to determine if the json path we want to change was defined in the global configuration file. Thus, the global configuration file serves as a “declaration file”, determining which json values exist and can be changed. It also compares the type of the value in global_configuration with the type contained in new_value. If it doesn’t match, you’ll get an exception. This, again, prevents you from accidentally writing 10.0 to the player-name variable, which would result in an error when re-loading the user config file at the next application start.

However, you should consider modify_user_value a low-level function. This is indicated by the fact that the last parameter is a value, not an arbitrary type. This means that you cannot pass a std::vector<int> to this function, you have to convert it into a json type, first (the convert_to function does this for you).

As a higher-level construct, sge provides user_config_variable, which is a template class:

template<typename T>
class user_config_variable
{
public:
	user_config_variable(
		json::object const &global_configuration,
		json::object &local_configuration,
		json::path const &);

	void
	value(
		T const &);

	T const &
	value() const;

	fcppt::signal::auto_connection
	change_callback(
		function<void(T const &)>);

	~user_config_variable();
};

If the fcppt::signal type looks strange to you, consider reading part III of the sge tutorial. I’ll assume that you know what signals are and how they’re used.

So what’s the idea behind user_config_variable? I’m going to answer this question and, in addition, I’m going to explain how to put together all the json functions presented so far to create a json configuration system for an application. The idea behind user_config_variable is illustrated best by the use case that made me write this class:

In the game fruitcut, which I’m currently working on, I’ve got a configuration variable “music volume” which is a floating point value in the range [0,1]. This variable is read from the global configuration file, but it may be changed by the user in the game’s main menu (via a slider). The change should, of course, be permanent, so I need to write the volume back to the user configuration file when the application exits.

The class responsible for music is called music_controller. This class, however, doesn’t contain any json configuration code, since it should be as reusable and abstract as possible (and maybe someone wants to use it in conjunction with his configuration system which isn’t based on json). The class has a method volume to set the volume. The initial volume is given to it in the constructor.

Looking at the class user_config_variable, the following steps have to be done to create a configuration system and to integrate a “writeable” configuration variable:

  1. Read in the global configuration file, store it in global_config_temp.
  2. Read in the user configuration file, store it in user_config.
  3. Merge the global config, the user config and the command line parameters, store the result in global_config. This will be the main database which we query when we need a config value, since it contains all the information the user and the system gave us.
  4. Create a user_config_variable called music_volume (with T = float), pass it global_config, user_config and the correct path.
  5. Create the music_controller, pass music_volume.value() to it. That’s the volume specified in the merged global_config, so it might come from the global config, the user config or the command line.
  6. Connect the music_controller::volume(...) function to the music_volume::value() getter via change_callback.
  7. Modify music_volume when the slider in the main menu is updated.
  8. user_config_variable will update the user_config tree in its destructor. So, after the variable is destroyed, you’re free to write back user_config to a file using output_tabbed (see above).
Advertisements
This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s