Savefile Character / Map Editor v1.0.1 (April 15, 2014)

Ask questions, share hints or chat in general about Eschalon: Book I.
User avatar
xolotl
Lieutenant
Lieutenant
Posts: 777
Joined: August 21st, 2008, 1:54 pm

Re: Savefile Character / Map Editor

Post by xolotl »

Jedi_Learner wrote:Here are some screenshots of what I'm working on.
Oh, sweet. Really looking forward to this. :)
User avatar
SpottedShroom
Captain Magnate
Captain Magnate
Posts: 1372
Joined: June 4th, 2010, 6:18 pm

Re: Savefile Character / Map Editor

Post by SpottedShroom »

I finally got around to messing with the editor in a non-trivial way today, and so I thought I'd share my first original content: the Monsterator. Ever wondered who would win in a fight: a taurax or a barrea mercenary? How about a dozen of each? Or throw some hive queens in there!
monsterator.png
monsterator.png (112.06 KiB) Viewed 14510 times
To use it, unzip monsterator.zip and place its contents in any save game slot. The entrance is in the second floor of the lighthouse, which is an empty map, so installing my version shouldn't disrupt existing games. Of course, your mileage may vary, so back up any saves you care about first!
Attachments
monsterator.zip
(138.39 KiB) Downloaded 605 times
User avatar
SpottedShroom
Captain Magnate
Captain Magnate
Posts: 1372
Joined: June 4th, 2010, 6:18 pm

Re: Savefile Character / Map Editor

Post by SpottedShroom »

Yeah, I put a couple of ideas for custom items in there as well. The healing potato was inspired by coin-op magic items in a buddy's D&D game. The other is a ruby that damages you but gives you ogre strength.

Oh, and I suppose I should have said - the lighthouse level 2 map doesn't look any different, but observant characters may discover otherwise.
laurelt
Pledge
Posts: 1
Joined: October 3rd, 2010, 6:15 pm

Re: Savefile Character / Map Editor

Post by laurelt »

Hi,

I am new to Eschalon and new to the editor.

I am on a Mac running OS X 10.6.4 and Parallels with Windows XP

I installed the editor on the windows machine, I used the .exe file just like it said to on the website and I get this error when I try to run it...

Unable to execute file:
C:\Program Files\Eschalon Utilities\eschalon_b1_char.exe

CreateProcess failed; code 14001.
This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.


I uninstalled and reinstalled twice. No dice. Am I doing something wrong?

Thanks! I was going to bring my save files over from the mac, edit them on the PC and then drag them back over.
User avatar
xolotl
Lieutenant
Lieutenant
Posts: 777
Joined: August 21st, 2008, 1:54 pm

Re: Savefile Character / Map Editor

Post by xolotl »

laurelt wrote:Unable to execute file:
C:\Program Files\Eschalon Utilities\eschalon_b1_char.exe

CreateProcess failed; code 14001.
This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.
Huh, weird. I'm actually totally unfamiliar with Parallels; is that some kind of VM type thing? Are you actually running Windows in a separate process, or is it some kind of emulation layer like Cider?

I did have a couple of people come to me with errors which turned out to be because they needed either XP Service Pack 3, or a Visual C++ Redistributable installed. (They had previously been on XP SP2.) The error they had was something different from yours, but perhaps it's related and just weird because of that Parallels environment you mentioned. If you're not on SP3 you may want to upgrade to that, or try installing the VC++ Redistributable. There's various versions of it (one from 2008 and another from 2010), and I'm not sure which one would necessarily do the trick. For 32-bit windows, the links would be: 2008 SP1, 2010. I'm sure the 64-bit versions wouldn't be too hard to track down.

Anyway, let me know if that takes care of it, or if there's anything else funky in your environment which might get in the way.
User avatar
xolotl
Lieutenant
Lieutenant
Posts: 777
Joined: August 21st, 2008, 1:54 pm

Re: Savefile Character / Map Editor

Post by xolotl »

Hey all - I just got a report from someone who got these utilities working on OSX, which may be good news for other Mac folks out there. I've put up a list of instructions at the project webpage. Let me know if that works for you! Also, if anyone feels like figuring out how to package it up into a .dmg or something, that would be pretty great, too.
DewiMorgan
Initiate
Posts: 13
Joined: November 1st, 2010, 1:20 pm

Re: Savefile Character / Map Editor

Post by DewiMorgan »

Bought E1 on steam a while ago, just recently started playing.

Got quickly annoyed with it, so decided to hack the savegames to cheat. This is usually a bad sign: once I start hacking a game, it's running low on my playability meter, at which point it's not long left on my HDD. The exception is easily-script-moddable games, like Bethsoft stuff and Furcadia.

So I quickly noticed that there was scripty stuff in the savegames, and maps. I had figured most of the obvious parts of the char file, listed out the values for a lot of the list-values, etc. But this morning I thought "this format's so open, I'm sure someone else must have documented this stuff - I'm reinventing the wheel here". However, when I Googled it, I never expected to find something THIS impressive!

What I did last night is just a tiny subset of what you've found already. I spotted a couple tiny bits where my notes differ from or add to what's on your scripting page for E1, though, since I ran "strings" on the exe file itself. Make sure to include unicode, since the commandlist in there is in 2-byte unicode. Most scripts seem to be ascii though.

So, those differences, I describe below.

activate_qt N - I'm assuming the param is 1 for the first in the list, and so on down?

effect (name) N - Here's a list of effect names I've seen in various contexts. Where I've seen them used with this command, the values are given in braces. No others are tested to work:
- Air Shielded
- Blessed
- Cat's Eyes (2)
- Chameleon
- Charmed
- Elemental Armor
- Enchanted Weapon
- Enkindled Weapon
- Entangled
- Gravedigger's Flame
- Greater Protection (3)
- Haste (I=2, II=4 III=6)
- Invisible (I=2, II=4 III=6)
- Keensight (3)
- Leatherskin (I=2, II=4 III=6)
- Mana Fortified (3)
- Nimbleness (3)
- Off-Balance
- Ogre Strength (3)
- Paralyzed
- Poisoned
- Predator Sight (3)
- Reveal Map
- Scared
- Stoneskin (2)
- Stunned

all_quests - not tested. Debugging command?

areacheck - Your writeup says it's not used in book 1, but I see it used in two places in scripts in the exe. This one I assume is a script that accompanies the endgame text:

Code: Select all

add_gold 83510 ; screen_fade_out ; map_port 52 ; move_player 7018 ; screen_fade_in ; areacheck
updatezones - This command is often included with the "areacheck" command, whatever that does. I'm assuming they both rationalise the map in some way when you teleport, such as this one from a conversation script:

Code: Select all

screen_fade_out ; map_port (Darkford) ; updatezones ; areacheck ; screen_fade_in
map_port - Contrary to your notes, the only place I've seen a any string parameter that isn't in (braces) is where the string is at the end of a script, so it's terminated by the \r\n string terminator, and hence (I assume) doesn't need the braces to delimit the string. I've seen map_port with both numeric map number params like "map_port 52", and with params in parentheses, like the above "map_port (Darkford)".

book N / learn_book N
I came up with this list, which I suspect is not exhaustive. Note that the parameters for book numbers and the learn_book numbers are NOT the same, though they are both called "booknum" in your writeup. Learn_book numbers are not the skill number either - they're the second number in the list below. Somewhere in the savegame, there'll be a set of fields (maybe a bitfield?) that stores which books have been "learnt", so you can't learn them twice. Not found this, but I suspect the sfx bitfield. 15 books, so 15 or 16 bits or bytes or ints, somewhere.

Book list:
1 The holy Tasslar
2 Of Men And Giants
3 Sealed Note
4 The Alchemist's Cookbook I
5 The Alchemist's Cookbook III (yes, out of order)
6 The Alchemist's Cookbook II
9 Torn Page
10 Journal Entry
13 Secrets of Transference Circuits
14 2nd Sealed note
15 37 The Legendary Swordsman
16 35 Keyholes and Tumblers
17 36 Bones to Splinters
18 32 the Greatest Ranger
19 41 Embrace the Night
20 38 Elements of Magick
21 43 Magicka Divine
22 39 Mapping Your World
23 42 Hidden Dangers and Treasures
24 40 Buying and Selling
25 34 Light Armor Field Guide
26 44 Heavy Armor Field Guide
27 33 It's a Trap!
28 31 The Art of Brewing
29 3rd Sealed note
30 Bloodstained Note
33 30 The Lumberjack
34 Imbuing Your Equipment
35 A History of Thaermore
36 Letter to Penelope
37 Mercenary's Note
38 The Goblins of Thaermore
39 The Orakur Are Watching
40 The Crimson Sky
41 Benny and the Dragon
42 Incriminating Letter
43 A World of Wonder
44 The Jewel of Thaermore
46 Sam's Guide to Demon Oil
48 Portal Instructions

cond_not_quest N - This is from 24.map: "cond_not_quest 25 (-1) ; trigger_talk 77" - I'm thinking that this shows two interesting things. First, like strings, negative numbers need to be quoted/delimited with braces, at least if they aren't the last parameter (which may mean that more commands can have negative numbers than previously thought?). Secondly, some kind of inactive quests (unstarted? killed? both?) seem to have a value of -1, so this handy code tests for that.

npc_disp_change 59 2 - change NPC 59's disposition to 2. I think 2 is attack, but I've only seen it on Lilith's (59) and Grimulk's (87) conversation options, when they go hostile. I've also seen 0 for npc 72, which also seems to be attack? Your text only lists 0 (hostile) and 1 (friendly), so I wonder what 2 is.

poison 200 - poison with this power level. I suspect 200 means "incurable".

npc_die 58 - kill the numbered NPC.

rename_item (SRC NAME) (DEST NAME) - rename an item (or, change all of an item to another type?). Since this is in conversation scripts, it probably works globally, renaming the item type. But it might just work on the PC's inventory. It's only used with unique, quest items, though, with a check to see that the player has them, so can't tell.

screen_fade_in - go from faded to black, to show play screen again. Used when teleporting, etc. No params.

screen_fade_out - fade play screen to black.

special_event 1 - trigger some special event? This may not be a command, as it's not included in the command list. But it looks like it is. May be a script ID or soemthing

strip_items - Remove all the player's items? Does this include script items? If so, are they placed in some chest? Takes no params, so it's not specifying a container. Only happens in conversation, so maybe the talking NPC gets the items? Used in one place:

Code: Select all

screen_fade_out ; move_player 5753 ; strip_items ; message (You wake laying upon cold dirt. The smell of death is all around you...) ; screen_fade_in

The following commands most likely are only usable from conversation scripts.

cleric_dehex - cleric remove curse action, from conversation options.

cleric_heal - cleric heal action, from conversation options.

init_trade (NAME) - trade with NPC NAME

rent_room 15 4690 - rent room from innkeeper dialog. I'm guessing price and room location? Values seen:
Porter: 15 4690
Eeru: rent_room 30 8959 (if you are polite)
Eeru: rent_room 50 8959 (if you aren't)

teach_skill (Garrett) 7 - Garrett teaches skill 7. Skill list seems to be the same as everywhere else:
01 Alchemy
02 Divination
03 Elemental
04 Light Armor
05 Heavy Armor
06 Shields
07 Cartography
08 Dodge
09 Hide in shadows
10 Lore
11 Meditation
12 Mercantile
13 Move silently
14 Pick Locks
15 Skullduggery
16 Spot hidden
17 Survival
18 Unarmed Combat
19 Bludgeoning weapons
20 Bow weapons
21 Cleaving weapons
22 Short bladed weapons
23 Swords
24 Thrown weapons


The following commands aren't used anywhere that I can find, but are included in the exe's list of commands. No indication of what params they support. The names are suggestive, but obviously they all need testing.

cond_detected - run following code if the player has been detected by foes?
convert - convert an item to another type, maybe?
curse - maybe works like "disease"?
delay - ?
full_restore - restore full mana? Or mana and health?
remove_barrier - what's a barrier?
spell - maybe works like "disease"?
trap - maybe works like "disease"?

There are two messages that suggest there may be cheat/debugging codes: "You've just learned every spell and became a Master of all Skills!" and "Reset this map?" - since these are right by "Press any key to continue" and "You decide to stand passively for a moment", I suspect they're input related, so probably cheat codes. There may also be some way to turn on "developer mode", since elsewhere there's the message "Developer Mode LOCKED OBJECT UNLOCKED!" - however, just as with the unused commands, there's no saying whether any of that was ever compiled in or made bug-free.

If any of that was useful, I can maybe poke around in the E2 string table, too, if that'd be helpful.

Other notes:

Code: Select all

            # More Unknowns
            for i in range(17):
                self.unknown.iblock1.append(self.df.readint())
            for i in range(5):
                self.unknown.ssiblocks1.append(self.df.readstr())
                self.unknown.ssiblocks2.append(self.df.readstr())
                self.unknown.ssiblocki.append(self.df.readint())
            self.unknown.extstr1 = self.df.readstr()
            self.unknown.extstr2 = self.df.readstr()
To me, this feels more like:

Code: Select all

            # More Unknowns
            for i in range(16):
                self.unknown.iblock1.append(self.df.readint())
            for i in range(6):
                self.unknown.ssiblocki.append(self.df.readint())
                self.unknown.ssiblocks1.append(self.df.readstr())
                self.unknown.ssiblocks2.append(self.df.readstr())
16 is a rounder number, and (number, string, string), seems more sensible to me.
So that gets rid of two unknowns for you: extstr1 and extstr2.


also:

Code: Select all

            for i in range(4):
                self.fxblock.append(self.df.readint())

            # An unknown, seems to be a multiple of 256
            self.unknown.anotherint = self.df.readint()

            # Character profile pic (multiple of 256, for some reason)
            self.picid = self.df.readint()

            # Disease flag
            self.disease = self.df.readint()

            # More Unknowns.  Apparently there's one 2-byte integer in here, too.
            self.unknown.shortval = self.df.readshort()
            self.unknown.emptystr = self.df.readstr()
feels more like:

Code: Select all

            # 5 byte bitfield.
            self.unknown.fxbits = self.df.readint()
            # Always 3F?
            self.unknown.fx3f = self.df.readbyte()

	    # 4 unknown ints
            for i in range(4):
                self.fxblock.append(self.df.readbyte())

            # An unknown.
            self.unknown.anotherint = self.df.readint()

            # Character profile pic.
            self.picid = self.df.readint()

            # Disease flag
            self.disease = self.df.readint()

            # More Unknowns.  Apparently there's a byte in here, too.
            self.unknown.byteval = self.df.readbyte()
            self.unknown.emptystr = self.df.readstr()

That is, rather than:
08 52 B8 1E
3F 0F 00 00
00 5F 00 00
00 5A 00 00
00 01 00 00 - int multiple of 256
00 00 00 00 - int profile pic multiple of 256
00 00 00 00 - int disease - is this x256 - see constantsb1.py
00 00 - short
0D 0A - string

To me, it looks like:
08 52 B8 1E - sfx/lighting bitfield?
3F - Always 3F?
0F 00 00 00 - int
5F 00 00 00 - int
5A 00 00 00 - int
01 00 00 00 - int 
00 00 00 00 - int profile pic
00 00 00 00 - int disease - need to edit constantsb1.py, see below.
00 - byte
0D 0A - string

That then changes constantsb1.py's code from:
    diseasetable = {
            0x0200: 'Dungeon Fever',
            0x0400: 'Rusty Knuckles',
            0x0800: 'Eye Fungus',
            0x1000: 'Blister Pox',
            0x2000: 'Insanity Fever',
            0x4000: 'Fleshrot',
            0x8000: 'Cursed'
        }
to:
    diseasetable = {
            0x02: 'Dungeon Fever',
            0x04: 'Rusty Knuckles',
            0x08: 'Eye Fungus',
            0x10: 'Blister Pox',
            0x20: 'Insanity Fever',
            0x40: 'Fleshrot',
            0x80: 'Cursed'
        }
No idea whether any of these help you decode the structure any, though.
User avatar
xolotl
Lieutenant
Lieutenant
Posts: 777
Joined: August 21st, 2008, 1:54 pm

Re: Savefile Character / Map Editor

Post by xolotl »

DewiMorgan wrote:...
Oh, cool! Thanks for the notes; I'll integrate them in with my docs soonish; I've been meaning to implement some things for the next version of the editor but haven't actually taken the time to work on it for awhile now (damn Minecraft, damn Fathamurk...) Anyway, I appreciate it; especially the work you've put into deciphering some of those unknown values. It's been awhile since I've looked at those.

Thanks again!

(Oh, and at this point I'd be remiss if I didn't thank SpottedShroom for his additional scripting notes as well, sent to me seemingly ages ago. I'll get those integrated soon too, I swear. :)
DewiMorgan
Initiate
Posts: 13
Joined: November 1st, 2010, 1:20 pm

Re: Savefile Character / Map Editor

Post by DewiMorgan »

I just tried actually installing and running the character editor and noticed that... well... you can't actually use it. At least, not for what I intended, which was to give myself a bunch of skill books and see what changed in the save file when I read one.

Far as I can tell, you can't just select an item type to plonk into a slot: you have to manually edit every field. Bleegh. But I might be missing something in the UI.

In the exe file, beginning around 13EF9C, there's a "master_item_sheet.csv" table of all the default items in the game.

Might be worth having your code parse that and use it to give users a picker where they can grab any item prefab they want.

Since the exe may change with updates, I'd not rely on the address, but rather just scan forward for the table header, which is the string: "ITEM CATEGORY","DESCRIPTION","LBS","SKILL REQUIRED","RARE","ICON","$$","COMB","DAM","AR","ATTR","AT m","SKILL","SK m","HP m","MANA m","2H m","DMG m","AR m","RES m","Effect","Script (called when USED)".

But ideally, there will be a better way to extract the embedded resources in the exe. Right after that table is the string "master_item_sheet.csv" (filename is in unicode, so search for i\x00n\x00c\x00... etc) - which I assume is the name of the original embedded file that contained that file.

If you can extract.... you can maybe embed, too. And that will give your editor control over conversation trees, and book contents, and so much more. Translations. Grammar/spelling fixes. New NPCs. You name it.

Immediately after that in the file is a bunch of monster stats in CSV, with the filename "incbin/entities.csv" (handy for making a monster picker in the map editor! Or if you can edit it, making new monsters!), then a bunch of texts, with the filename "incbin/books.txt" - very handy if the user is picking a book, maybe? And if you can edit it, for making books!

After that is a bunch of conversation tree files, each called something like "incbin/87.tre". Decoding the conversations could be fun, as they contain scripts too. But making it so that new .tre files can be created, that's the ultimate.

Hey, Googling "incbin blitzbasic" tells me this is a standard thing with BB. So, maybe we can extract, modify, and reinsert. However, can't find anything on their forums on extracting, other than a link to a program that extracts standard format files that are embedded in other files, but isn't any use to us, since it doesn't have plugins for any of the filetypes we're interested in.

I guess, just need to do it the hard way then. Need to OK it with BG of course. But they've been good about the editor so far.

These are the ones that seem to be in the exe:
incbin/51.tre
incbin/52.tre
incbin/53.tre
incbin/54.tre
incbin/55.tre
incbin/56.tre
incbin/57.tre
incbin/58.tre
incbin/59.tre
incbin/61.tre
incbin/62.tre
incbin/63.tre
incbin/64.tre
incbin/65.tre
incbin/66.tre
incbin/67.tre
incbin/68.tre
incbin/69.tre
incbin/70.tre
incbin/71.tre
incbin/72.tre
incbin/73.tre
incbin/74.tre
incbin/75.tre
incbin/76.tre
incbin/77.tre
incbin/79.tre
incbin/80.tre
incbin/81.tre
incbin/82.tre
incbin/83.tre
incbin/85.tre
incbin/86.tre
incbin/87.tre
incbin/88.tre
incbin/89.tre
incbin/90.tre
incbin/91.tre
incbin/92.tre
incbin/93.tre
incbin/94.tre
incbin/95.tre
incbin/96.tre
incbin/books.txt
incbin/celtic.ttf
incbin/entities.csv
incbin/f1.ttf
incbin/kelt.ttf
incbin/master_item_sheet.csv
incbin/narrative.txt

Filename seems to come AFTER the body of the file, and is in unicode.

The incbin data seems to be after all other executable content in the exe, so editing these files MAY be as simple as figuring out where the length byte is before each file, and changing that.

Some random ponderings:
Is the character "$" translated by message(str) to the player's name? Or is that just conv trees? Does prefixing a word with * ^ & or $ make it do anything special, or is that just in skill descs?

What are the numbers from the commands "narrative N" and "notebox N" from?

Also, do we know what affects the verb? That is, whether you EAT, DRINK, READ, or USE an item that has a script? Is that governed by the item type?

I'm only writing all this because I'm slightly delusional from a cold, waiting for my fever to break so I can sleep. I'm too out-of-it to do any real work, I can't concentrate at all, but flow-of-consciousness poking at a game seems quite possible and is better than lying awake looking at the ceiling. So, once I'm better, odds are I won't post again.

Kill_quest displays the message "A quest in your Quest Journal has been removed because it can no longer be finished. You may still be able to complete the game." - so if you want to silently disable a quest, better to set it to stage -1?

Ooh, turns out you can have custom ports. Didn't see that mentioned in the manual, but yeah - stick "mypic.png" (60x60) in the main app folder, and it'll pick it up next time you create a character. And, hey, the character editor already knows about those and respects them! Nice. Way ahead of me as usual.

5:30am. Off to bed.
User avatar
xolotl
Lieutenant
Lieutenant
Posts: 777
Joined: August 21st, 2008, 1:54 pm

Re: Savefile Character / Map Editor

Post by xolotl »

DewiMorgan wrote:I just tried actually installing and running the character editor and noticed that... well... you can't actually use it. At least, not for what I intended, which was to give myself a bunch of skill books and see what changed in the save file when I read one.

Far as I can tell, you can't just select an item type to plonk into a slot: you have to manually edit every field. Bleegh. But I might be missing something in the UI.
That's largely right - if you expand the thingy next to the items in the main list, you do have the ability to at least copy/paste items, so you can clone items pretty easily and then just modify the single fields that you're interested in. I would be the first to agree that the UI is hardly ideal; I'm a commandline guy at heart, and I would never list UI design amongst my strengths. :)
In the exe file, beginning around 13EF9C, there's a "master_item_sheet.csv" table of all the default items in the game. Might be worth having your code parse that and use it to give users a picker where they can grab any item prefab they want.
Ah, nifty. I've specifically avoided trying to read anything out of the actual game executable, though, and I'm pretty unlikely to do so in the future. I'd have to maintain an understanding of the executable on three different platforms (linux/mac/win), and then Windows of course is bundled by different content distribution networks as well, which could change things around. In the end I'm just not interested in maintaining that kind of thing (and I wouldn't want to distribute an extracted CSV with the util; I'd like to keep the editor as "clean" as possible).

As you mention, I could certainly scan for a header, but I'm afraid I just don't have much interest in doing so. :)

I have occasionally considered just making up my own item prototypes that you could select from a dropdown or something, and that may end up happening one of these days. Perhaps I'll code up a simple class that could be used to store premade objects and call out for submissions, rather than doing all the drudgework myself.
If you can extract.... you can maybe embed, too. And that will give your editor control over conversation trees, and book contents, and so much more. Translations. Grammar/spelling fixes. New NPCs. You name it.
Yeah, though once again, I don't have any interest in my util parsing content in the executable itself, and I don't want to redistribute content embedded in there, either. (Not to mention that actually editing that information would be problematic at best; I'd only really be able to do a readonly view of those things, which doesn't strike me as worth the effort.) For what it's worth, BW has mentioned a few times that he'll probably release his own game editing tools when Book 3 comes out (or at least at some point after Book 3 is out), at which point we'd have a more comprehensive set of tools available to work on things like conversations, etc).
5:30am. Off to bed.
Get better! And thanks for the tips re: scripting.
DewiMorgan
Initiate
Posts: 13
Joined: November 1st, 2010, 1:20 pm

Re: Savefile Character / Map Editor

Post by DewiMorgan »

I'm pretty sure that for blitz-basic, platform-dependent differences in the exe will be at worst a minor issue. I've confirmed the same basic exe layout for E1 in Linux, Mac and Windows. The script below works to extract the files and strings for any of them.

Differences between E1 and E2 are more likely to be an issue than differences between OSs. *checks* Yeah - E2 has nothing useful included in the exe - odds are the good stuff's in "datapak", which looks encrypted apart from the filelist at the end. Well, poo, that ruins that plan then.

Still, for anyone who cares, or is interested in learning from the scripts in the conversation trees, you can use this script to extract the embedded files and unicode strings in both E1 and E2 under any OS, though exe filenames may need changing. Script is php as my python-fu is weak. Run it in the same folder as your executable.

Code: Select all

<?php
$filenames = array('book_2.exe', 'eschalon_book_1.exe', 'eschalon_book_1', 'Eschalon Book I');
$fileFound = 0;
foreach ($filenames as $filename) {
	if (file_exists($filename)) {
		if ($filename == 'book2.exe') {
			$folder = 'resources';
		}
		else {
			$folder = 'incbin';
		}
		@mkdir($folder);
		$fileFound = 1;
		break;
	}
}
if ($fileFound == 0) {
	echo "Error: Could not find the executable.\n";
	exit;
}

# Every exe I've checked uses a different value for 'constant' :(
#                 pad    [ constant  ] [     maxint   ] [    stringlength   ] [    string    ]  pad
$unicodeRegex = '/(\x90*[\x00-\xff]{4}\xff\xff\xff\x7f[\x00-\xFF][\x00-\xFF]\x00\x00([\x20-\x7e]\x00)+\x90*)/';

$file = preg_split($unicodeRegex, file_get_contents($filename), -1, PREG_SPLIT_DELIM_CAPTURE);

$unicodeStrings = '';
$fileCount = $stringCount = 0;

for ($i = 0; $i < count($file); $i++) {
	if (!preg_match($unicodeRegex, $file[$i])) { continue; }
	// Clean unicode to ascii, primitively.
	$string = preg_replace('/^\x90*[\x00-\xff]{4}\xff\xff\xff\x7f[\x00-\xFF][\x00-\xFF]|\x00|\x90/', '', $file[$i]);

	if (substr($string, 0, strlen($folder)+1) == "$folder/") {
		if (preg_match('/\.ttf$/', $string)) { // Fix TTF leading nulls.
			$file[$i-1] = substr($file[$i-1], strpos($file[$i-1], "\x00\x01\x00\x00"));
		}
		file_put_contents($string, $file[$i-1]);
		$fileCount++;
	}
	$unicodeStrings .= $string . "\n";
	$stringCount++;
}
file_put_contents("$folder/unicodestrings.txt", $unicodeStrings);
echo "Exported $fileCount files to $folder/\n";
echo "Exported $stringCount strings to $folder/unicodestrings.txt\n";
Legal

The E1 software license says "* Limitations on Reverse Engineering, Decompilation and Disassembly. The Software in both object code and source code form includes valuable trade secret information of Basilisk Games. You may not make any copies of the Software beyond the number necessary to exercise your license rights in the Software, and shall not provide copies of the Software to any third party. You may not reverse engineer, decompile, or disassemble the Software or otherwise attempt to gain access to the source code for the Software. "

My reading of this is that program code is "source code for the software", and must not be touched, but embedded data is just data, fine to extract for personal use and modding, but obviously still subject to copyright. However, IANAL: consult one if concerned, or ask BG. In particular, scripts seem like a greyish area that might invalidate your license.

Etc

The above code assumes that the beginning of the file won't look like Unicode. That's true in the case of all the embedded files, but a more rigorous script would parse the "string length" int, and not grab anything more into the strings than it needed to. Because I was using regexes, I didn't bother, so a very few of the strings in the extracted UnicodeStrings.txt table have one or two garbage characters on the end, like: "Weapon, Thrown$", "You are cursed!!i", "response_marker.png$", etc.

Looks like I was WRONG about the embedded files being the last stuff in the exe, plus I can't find any allocation table for resources anyway, both of which mean that changing the length of any of this stuff will unfortunately be an unholy nightmare. Still, data within the files should be able to be changed freely, so long as the length remains. So conversation options COULD be added, and typos fixed, at the cost of editing some other conversation items to be shorter.

[Edit: Another assumption I made in the above scripts is that files are separated by unicode strings. This is only mostly true. They are terminated by unicode strings, and they come one after the other, but there's no leading marker. So, the very first file will have a few bytes of leading junk. Except, we know that the first file, in both cases, is a font file, and we know what sequence font files begin with, so we can strip the leading bytes with a reasonable degree of confidence. I just updated the code to do that properly on all OS's: Linux and Mac had more leading junk than windows, but this fix should resolve that.]

[Edit2: I'd been assuming that strings were 255 characters or less, which is false. Changed the assumption to 64k characters or less, which is true.]
Last edited by DewiMorgan on November 11th, 2010, 3:56 pm, edited 1 time in total.
DewiMorgan
Initiate
Posts: 13
Joined: November 1st, 2010, 1:20 pm

Re: Savefile Character / Map Editor

Post by DewiMorgan »

Decoding the .tre files
Here's what I've got, which isn't much. I write ints as [NN] for brevity, eg "0x0A 0x00 0x00 0x00" = "[0A]". Anywhere you see [nn], I mean a 4-byte int.

Each NPC has a .tre file named with the NPC's entity number. So Gunther (entity 66) has his conversation tree stored in the file "66.tre".

Conversation strings are stored as a series of string pairs, which I'll call "exchanges", where the string the player can say is immediately followed by the NPC's response.

The exchange is preceded by two ints.

The first, int1, is the exchange id, which can be anything so long as it's unique within the file. Typically, it starts at 0 and increments for each exchange, but it doesn't need to: they are sometimes randomly ordered in the file, with gaps. This is typically [00] for the initial greeting, which is played only once, when you first meet the character, and [01] for the subsequent greeting that you see any other time, though I suspect that these numbers are not necessary.

The second, int2, is the "category". This is [00] for the initial-meeting greeting (which always mentions the name of the character), and [01] for the subsequent-meetings greeting. All other values are taken from int11, described below.

Then come the two "conversation" strings. Either or both can be zero length. Each conversation string is stored as int stringlength, then the ascii characters, with no terminator, like "[04]Fred". So a zero length one is [00]. In my examples, I put [LL] to mean "I'm too lazy to calculate this".

In these strings, the \n character can be used on its own within a conversation string to break the line. This is most often seen as \n\n for a blank line. The $ character will be replaced by the player name (but only at display time: it counts as a single character for the string length int, of course!)

The exchange is followed by seven ints (which I've numbered int5 to int11), then four "data" strings. Unlike the conversation strings, these have no length int, and are terminated by \r\n.

The number in int5 is a quest number to test, or zero. Then int6 stores the quest stage to test against, or zero. If both are non-zero, then the exchange will only be available if the quest is at that stage. There is no case where only one of these ints is zero.

Int7 is unknown. It is only set in the files listed below. Other than zero, no value appears in more than one file. No file contains more than two values. Most values are used once only, but 14, 15, and 27 are used up to three times. Higher file numbers contain higher int7 numbers. The numbers do not usually correspond to any exchange id in the file.

Code: Select all

int7val : file : times found
06 : 58.tre : x 1
11 : 56.tre : x 1
12 : 56.tre : x 1
14 : 67.tre : x 2
16 : 67.tre : x 1
17 : 63.tre : x 1
20 : 83.tre : x 1
21 : 72.tre : x 1
22 : 73.tre : x 1
25 : 77.tre : x 3
27 : 80.tre : x 3
45 : 85.tre : x 1
46 : 85.tre : x 1
47 : 76.tre : x 1
49 : 86.tre : x 1
50 : 86.tre : x 1
Int8 is unknown. It is set in the same files as int7, except for 63 or 76. Values appear across multiple files, and multiple times in the same file. The numbers always correspond to an existing exchange id in the file, but this may be coincidence, as they are all low numbers.

Code: Select all

int7val : file : times found
1:67.tre:x 2
1:72.tre:x 1
1:77.tre:x 2
1:86.tre:x 1
2:67.tre:x 1
2:80.tre:x 3
2:86.tre:x 1
5:56.tre:x 1
9:56.tre:x 1
9:58.tre:x 1
9:73.tre:x 1
9:77.tre:x 1
9:83.tre:x 1
9:85.tre:x 1
12:85.tre:x 1
Int9 and int10 are always zero.

Int11 is the next conversational category to display options for. -1 sends the player back to the initial greeting, and is only used in exchanges where the NPC string is empty. If int11 points to a category that does not exist, then the conversation terminates. However, in this case, there is always an "empty" exchange (see below) with the same exchange ID as the non-existent category ID.

The first data string, str3, is before the first \r\n, and is an optional item name, which I believe is the name of an item that needs to be in the PC inventory in order for the exchange to be offered as an option in the list. This is typically used when removing items:

Code: Select all

[08][02][LL]"I got your thingy [Give Thingy]
[LL]"My thingy!"
[18][01][00][00][00][00][09]
Thingy\r\n
remove_item (Thingy) ; add_gold 10 ; quest 24 9\r\n
\r\n\r\n
With the rename_item command, this seems occasionally used as another way of tracking quest statuses, if another variable is needed instead of the the normal quest level mechanism.

As shown above, any script for the exchange is placed in the second data string, str3, before the next \r\n. The script is run before the NPC response is displayed, so the following exchange is displayed in a sensible order:

Code: Select all

[02][02][LL]"Yes, I'd like to have a look at your inventory."
[LL]"Thanks for your patronage"
[00][00][00][00][00][00][03]
\r\n
init_trade (gunther)\r\n
\r\n\r\n
The next two data strings, str5 and str6, are always empty. Possibly one is for a script that runs after the NPC's text displays?

Since the only required part of the exchange is the incrementing id, you can and do see 'empty' exchanges like this. One of these exists for every conversational category that terminates a conversation, with its category id set to 0 and its exchange id set to the category id. I don't know why:

Code: Select all

[03] - int1=id (ids of [00] are initial greetings, so won't be blank.)
[00] - int2=no category
[00] - int3+str1=zero length PC conversation string
[00] - int4+str2=zero length NPC conversation string
[00] - int5=no quest to check
[00] - int6=no quest stage to check
[00] - int7=unknown
[00] - int8=unknown, reference to another int1?
[00] - int9=unknown, always [00]
[00] - int10=unknown, always [00]
[00] - int11=no subsequent category
\r\n - str3=no item needed
\r\n - str4=no script to run
\r\n\r\n - str5+str6=empty data strings
So here's a full conversation tree:

Code: Select all

Initial greeting: A man waves. "Hi, I'm Joe." 
Subsequent greeting: Joe looks up and waves.

Then displays bottom level of this conversation tree.

"Train me!" / (training happens) "Thanks."
 - "[Back]" / ""

"Trade with me!" / (trading happens) "Thanks."
 - "[Back]" / ""

"Tell me about yourself." / "I'm boring."
  - "Nevermind, then." / ""
  - "Tell me anyway." / "I'm REALLY boring."
    - "Nevermind, then." / ""
    - "No really, tell me." / "Once upon a time...."
      - "That was gripping!" / ""

(if not already told to get a fish)
"Give me a job!" / "OK, get me a fish."
  - "OK, brb" / ""
    - end convo.
  - "I've got one on me." / "Great."
    - "[Back]" / ""

(if told to get one, and has one)
"I've got you a fish." / "Great."
  - "[Back]" / ""

"I have to go."
  - end convo.
That becomes:

Code: Select all

[00][00]
[LL=00] - empty PC string for greetings.
[LL]A man waves. "Hi, I'm Joe."
[00][00][00][00][00][00][02] - display cat 2.
\r\nquest 63 0\r\n\r\n\r\n - initial greeting zeroes the quest, just in case.

[01][01]
[LL=00]
[LL]Joe looks up and waves.
[00][00][00][00][00][00][02] - display cat 2.
\r\n\r\n\r\n\r\n

[02][02]
[LL]"Train me!"
[LL]"Thanks."[00][00][00][00][00][00][03]
\r\nteach_skill (Joe) 24\r\n\r\n\r\n

[03][03]
[LL][BACK]
[LL=00] - empty NPC string, because we're going back to the greeting.
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[04][02]
[LL]"Trade with me!"
[LL]"Thanks."[00][00][00][00][00][00][04]
\r\ninit_trade joe\r\n\r\n\r\n

[05][04]
[LL][BACK]
[LL=00]
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[06][02]
[LL]"Tell me about yourself."
[LL]"I'm boring."
[00][00][00][00][00][00][05]
\r\n\r\n\r\n\r\n

[07][05]
[LL]"Nevermind, then."
[LL]
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[08][05]
[LL]"Tell me anyway."
[LL]"I'm REALLY boring."
[00][00][00][00][00][00][06]
\r\n\r\n\r\n\r\n

[09][06]
[LL]"Nevermind, then."
[LL]
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[0a][06]
[LL]"No really, tell me."
[LL]"Once upon a time...."
[00][00][00][00][00][00][07]
\r\n\r\n\r\n\r\n

[0b][07]
[LL]"That was gripping!"
[LL]
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[0c][02]
[LL]"Give me a job!"
[LL]"OK, get me a fish."
[00][00][00][00][00][00][08]
\r\nquest 63 1 ; give_item (Fishing Rod)\r\n\r\n\r\n

[0d][08]
[LL]"OK, brb"
[LL]
[00][00][00][00][00][00][0e]
\r\n\r\n\r\n\r\n

[0e][00] - empty "end conversation" entry.
[LL]
[LL]
[00][00][00][00][00][00][00]
\r\n\r\n\r\n\r\n

[0f][08]
[LL]"I've got one on me."
[LL]"Great."
[00][00][00][00][00][00][09]
Fish\r\nquest 63 9 ; add_gold 10 ; remove_item (Fish)\r\n\r\n\r\n

[10][09]
[LL][BACK]
[LL=00]
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[11][02]
[LL]"I've got you a fish."
[LL]"Great."
[3f][01][00][00][00][00][0a] - [3f][01] means check quest 61 is stage 1
Fish\r\nquest 63 9 ; add_gold 10 ; remove_item (Fish)\r\n\r\n\r\n

[12][0a]
[LL][BACK]
[LL=00]
[00][00][00][00][00][00][-1]
\r\n\r\n\r\n\r\n

[13][02]
[LL]"I have to go."
[LL]
[00][00][00][00][00][00][14]
\r\n\r\n\r\n\r\n

[14][00] - empty "end conversation" entry.
[LL]
[LL]
[00][00][00][00][00][00][00]
\r\n\r\n\r\n\r\n
Now, I'm sure you can see scope for immediate optimisation there - all four "[Back]" blocks can be combined into one by giving them the same category ID, for a start. And those empty exchanges, same thing.

Except, this is never done. I don't know why: I assume the software they used to generate these files didn't consider the possibility of reusing exchanges for multiple responses.

But what that means is that there's plenty of space to fit in another conversation option or two into most conversation files.

OK, think I got all the procrastinatory marrow out of this bone that I can be bothered to get. Dropping it now.

[Edit: filled in the two unknown ints, so now we know how the trees work.]
DewiMorgan
Initiate
Posts: 13
Joined: November 1st, 2010, 1:20 pm

Re: Savefile Character / Map Editor

Post by DewiMorgan »

Found this in another thread:
BasiliskWrangler wrote:As long as no internal game information is being posted (such as charts or item lists) then I don't mind. Most people don't know how to use hex editors and if you want to risk corrupting your saved games to get a few stat boosts, that is your choice.

You cannot use a hex editor on the main executable or any other file that is part of the initial installation. That would be in violation of the license that you agreed to when you installed the game.
Well, dang. I was considering making a little something to allow conversations to be modded, but while that might've been fun, I feel that it's much more important to respect BW's restrictions on his own work.

I'll happily delete any/all my above posts that're against those restrictions, and am happy for them to be deleted by admins, too.
User avatar
xolotl
Lieutenant
Lieutenant
Posts: 777
Joined: August 21st, 2008, 1:54 pm

Re: Savefile Character / Map Editor

Post by xolotl »

xolotl wrote:Oh, cool! Thanks for the notes; I'll integrate them in with my docs soonish... (Oh, and at this point I'd be remiss if I didn't thank SpottedShroom for his additional scripting notes as well, sent to me seemingly ages ago. I'll get those integrated soon too, I swear. :)
Well, I posted that on November 1 last year, so it seems only fair that I should actually follow through on my promise before we get to a whole year's delay. The suggestions from both folks have now been integrated into the scripting docs on my site.

In other news: hello, everyone! I've not been entirely absent over the past year - I've been popping in every month or two to catch up on threads - but I have definitely been letting these utilities fall by the wayside. I had ended up maintaining a utility for Minecraft which has pretty much sucked up all the time that I'd have been spending on these utilities. I've been meaning to hop back in here, though, and some recent activity in the Book II editor thread seems to have spurred me into action.

In addition to integrating the extra scripting notes (and hopefully making a few more changes before I push out a new release of the utilities) there's a couple of other changes. One is that I've set up a Sourceforge project here: https://sourceforge.net/projects/eschalonutils/ - The "main" website for the utilities will remain as it always has (linky), but Sourceforge is where the actual file releases will be hosted, and it might be useful to some folks because of its RSS integration and all that (this, of course, assuming that I start maintaining this thing a little better than I have over the past year).

I've also moved the code repository from being a private, locally-hosted CVS thing on my own fileserver, to git. The sourcecode was always distributed inside the .tgz and .zip distributions, but now you can also find it at github: https://github.com/apocalyptech/eschalon_utils - I'm also pushing any changes to Sourceforge's repository, though I consider github to the the origin. One upswing of that is that anyone could have access to unreleased code if you want, including a pretty nifty script editor component which, it turns out, hadn't actually made it into a real release yet (whoops!)

So yeah, that's that. Hopefully I should have another release of this stuff out within a week or so!
User avatar
SpottedShroom
Captain Magnate
Captain Magnate
Posts: 1372
Joined: June 4th, 2010, 6:18 pm

Re: Savefile Character / Map Editor

Post by SpottedShroom »

Good to see you're still making improvements to the editor. Please allow me to make an unreasonable feature request. I'd love to see an API into the map interface, so that I could write code like this:

Code: Select all

import eschalon

m = eschalon.Map()

m.draw_wall(10, 10, 10, 20, WALL_BRICK)
m.draw_wall(10, 20, 20, 20, WALL_BRICK)
m.draw_wall(20, 20, 20, 10, WALL_BRICK)
m.draw_wall(20, 10, 10, 10, WALL_BRICK)
for i in range(10, 20):
  for j in range(10, 20):
    m.setFloor(i, j, FLOOR_BRICK)

m.save('somefile.map')
Post Reply