Idx and pkg structure. Extract EdenEternal, Kitsu Saga, Grand Fantasia

Idx file structure. Valid for Eden Eternal, Kitsu Saga, Grand Fantasia. Can be used for extracting resources from pkg archives.

Code:

---

//--------------------------------------
//--- 010 Editor v3.2.1 Binary Template
//
// File: EdenEternal, Kitsu Saga, Grand Fantasia pkg
// Author: Dwar & Genz
// Revision: 2011-08-07
// Purpose: Resource unpack
//--------------------------------------
struct IdxSignature
{
    uint Dummy[0x41];
    char Signature[0xC];        //idx signature
    uint Unknown2;
    uint Unknown3;
    uint Unknown4;
    uint Unknown5;
    uint Unknown6;
} Signature;

struct FileHeader
{
    uint FileID;              //file id, or file number
    uint Offset <format=hex>;  //offset of packed file in pkg
    uint Unk4;
    uint SizePacked;          //size of packed file
    uint Unk6;
    uint Unk7;
    uint Unk8;
    uint Unk9;
    time_t PackTime;
    uint Unk11;
    time_t OpenTime;
    uint Unk13;
    time_t ChangeTime;
    uint Unk15;
    uint SizeOriginal;        //original size of packed file
    char FileName[0x104];
    char FilePath[0x104];
    uint Unk1;
    uint pkgNum;              //pkg sequence number
    uint FileCRC <format=hex>;
};

while (!FEof())
    FileHeader File;

---

Additional

pkg.idx format

Code:

---

Offset    Bytes  Description
0x0000          72          Unknown
0x0048          N×592          Records (see below)
0x0004          4          Unknown. My guess is a checksum.

---

Record format

Code:

---

Offset    Bytes    Type            Description
0x0000          4          Unknown
0x0004          4          Integer          File counter
0x0008          4          Pointer          Package offset, in the corresponding pkg-file.
0x000C          4          Pointer          Index offset, in this file. I don't know what this is for. My guess would be some deprecation mechanism.
0x0010          4          Integer          Size of compressed data.
0x0014          16          4×Integer          Unknown. Sometimes parts are zero, and sometimes seems to contain random data.
0x0024          8          Integer          Timestamp 1. Not sure what each of these represents.
0x002C          8          Integer          Timestamp 2.
0x0034          8          Integer          Timestamp 3.
0x003C          4          Integer          Unpacked size.
0x0040          260          String          Filename.
0x0144          260          String          Path. Ends with a backslash.
0x0248          4          Unknown          Always zero.
0x024C          4          Integer          Package number. 1 = pkg001.pkg etc.

---

BMS Script

Code:

---

# Aeriagames pkg.idx/pkg???.pkg
#  Eden Eternal
#  Kitsu Saga
# script for QuickBMS
# quickbms . aluigi . org

getdstring DUMMY 260
getdstring SIGN 32
math PKG_OLD = -1
get FULLSIZE asize
do
    get DUMMY long
    get OFFSET long
    get DUMMY long
    get ZSIZE long
    getdstring DUMMY 0x28
    get SIZE long
    getdstring NAME2 260
    getdstring NAME 260
    get DUMMY long
    get PKG long
    get DUMMY long
    if PKG != PKG_OLD
        string PKG_NAME p= "pkg%03d.pkg" PKG
        open FDSE PKG_NAME 1
        math PKG_OLD = PKG
    endif
    string NAME += NAME2
    clog NAME OFFSET ZSIZE SIZE 1
    savepos CURR
while CURR < FULLSIZE

---

Python script to extract the packages. Run it in the Grand Fantasia folder and it will put all the files into a sub-folder named "out". Requires Python v2.5.2+

Code:

---

from __future__ import with_statement
import sys
import os
import zlib
import struct
from operator import itemgetter

class Record(tuple):
    def __new__(cls, *result):
        if len(result) != 17:
            raise ValueError("Requires 17 arguments. Recieved %d" % (len(result,)))
        return tuple.__new__(cls,result)

    unknown00 = property(itemgetter(0))
    filenum = property(itemgetter(1))
    package_offset = property(itemgetter(2))
    index_offset = property(itemgetter(3))
    package_bytes = property(itemgetter(4))
    unknown05 = property(itemgetter(5))
    unknown06 = property(itemgetter(6))
    unknown07 = property(itemgetter(7))
    unknown08 = property(itemgetter(8))
    timestamp1 = property(itemgetter(9))
    timestamp2 = property(itemgetter(10))
    timestamp3 = property(itemgetter(11))
    unpackedsize = property(itemgetter(12))
    name = property(itemgetter(13))
    path = property(itemgetter(14))
    unknown16 = property(itemgetter(15))
    pkgnum = property(itemgetter(16))

def read_index():
    index_byte_size = os.stat('pkg.idx')[6] # st_size

    with open('pkg.idx','rb') as f:
        header = f.read(288)
        data = f.read(index_byte_size - 288 - 4)
        checksum = f.read(4)

    if len(data) % 592 != 0:
        raise IOError("Invalid filesize: Not a multiple of %d plus %d" % (size, 288 + 4))

    files = parse_index(data)

    return header,files,checksum

def parse_index(data):
    format = '<9L3QL260s260s2L'
    size = struct.calcsize(format) # 592 bytes
    files = []
    for offset in xrange(0,len(data),size):
        record = list(struct.unpack(format,data[offset:offset+size]))
        record[13] = record[13].partition('\0')[0]
        record[14] = record[14].partition('\0')[0]
        record = Record(*record)
        files.append(record)
    return files

file_cache = {}
def unpack(pkgnum,offset,size,filename):
    pkgname = "pkg%03d.pkg" % (pkgnum,)
    f = file_cache.get(pkgname)
    if f is None:
        f = file_cache[pkgname] = open(pkgname,"rb")
    f.seek(offset)
    data = f.read(size)
    data = data.decode('zlib')
    filename = os.path.join('out',filename)
    path,name = os.path.split(filename)
    if not os.path.isdir(path):
        os.makedirs(path)
    with open(filename, 'wb') as f:
        f.write(data)


headers,files,checksum = read_index()

n = len(files)
sort_key = lambda o: (o.pkgnum,o.package_offset)
for i,rec in enumerate(sorted(files,key=sort_key)):
    fn = os.path.join(rec.path,rec.name)
    print '(%6.2f%%) Unpacking %s...' % (100.*i/n,fn)
    unpack(rec.pkgnum,rec.package_offset,rec.package_bytes,fn)

---

May be useful :)

Hi,
i used the Python script for the unpacking of Grand Fantasia, but for repack the file?

Generally, to pack again you should fully repeat structure of original archive. As you can see, in fileheader we have lots of Unknown fields, so if you can define them, you can make a proper packer

hi,

thanks for providing this informations.
Used it to implement a cli gdo patcher.

great work, and thanks for sharing.

Hello all,
Right now i'm testing the bms script but i've some error :

Error : invalid operator p

Can i have some help too solve this problem?

Hello,

I was really happy to see that you provided the codes for us, who cannot download the extractor file, to use.
My problem is that I am only a beginner with a little knowledge about programming this advanced, to the extent that I can't even tell what languages you used...

After searching through Google's resources and making an attempt to use Pyhton, that I failed miserably, I tried using quickbms. This is what I did:
I saved the provided code as a .txt file.
Run the quickbms.
Chose that .txt file a the script plugin.
Chose the target .pkg file, and extract directory.
And this is when I failed again. There was an error: error in src\file.h line 213: myfopen()
Error: No such file or directory.

This is my first time using the quickbms so I am clueless about how to solve this problem. I virtually have no idea even what this means.

You guys made such a great job writing those codes and then giving us them. I really feel bad about asking for so much help, but could you give me some hints?

Another thing, that I should probably ask at the very beginning. Does this program (codes) extracts only sound and images? Or can be used to extract game data, such as algorithms for some processes?

I am sorry if asking those questions is out of place.

Thank you :)

The only way I managed to extract pkgs is with quickbms, without any error. But I don't know how to look up and define the empty parameters the files are supposed to have so I can repack with that python script.

Btw, I did not use any pkgs of these games named above, I was extracting the new chibi MMO Aura Kingdom. I'm just trying to figure out how to repack now.

Hey, I'm french so not good english '-'
A friend give me a very useful file because it reimported 4 files in Grand Fantasia. But I don't have very well understand the things^^. And now, I've unpack pkg.idx and I found a file (R831.kf). I want to re-import this file like the 4 files but I don't now how I must make :s
I give you the file who reimported the 4 files and I hope you tell me how reimport R831.kf with the same thing ^^

Again sorry for bad english x')