How to set up a profile?

I don't know if setting up a profile is necessary for a Gecko app. Probably.
Anyway, to set up a profile you just need to provide the runtime with a directory for it.

Note: nsIFile is a cross-platform representation of a file system path (not a file itself). It can point to a file or a directory.

So first get the standard directory for application data on your operating system:
nsCOMPtr<nsIFile> appDataDir;
nsresult rv = NS_GetSpecialDirectory(NS_APP_APPLICATION_REGISTRY_DIR, getter_AddRefs(appDataDir));
(On Windows this will be "C:\Documents and Settings\\Application Data\Mozilla\".)

Append your application name to it:
appDataDir->AppendNative(nsCString("winembed"));

I don't know whether this is needed (probably not?):

nsCOMPtr<nsILocalFile>localAppDataDir(do_QueryInterface(appDataDir));

Create directory provider that will report where the profile dir is when asked:

nsCOMPtr<nsProfileDirServiceProvider> locProvider;
NS_NewProfileDirServiceProvider(PR_TRUE, getter_AddRefs(locProvider));

locProvider->Register();

And set the profile dir on this directory provider:
locProvider->SetProfileDir(localAppDataDir);

See code: xulrunner-1.9.1.3-source\mozilla-1.9.1\embedding\tests\winEmbed\winEmbed.cpp, function StartupProfile()