Archive Library¶

The Archive Library supports reading and writing files from the PC releases of the game.

File List contains one or more path strings. Each starting with C:\ff8\Data\ ending with \r\n. These are in the same order as the File Index. You scan the File List to find the items you want, while doing this you would increment a counter. The counter is used to know which entry to get from File Index. Lastly, you use the File Index to read from the File Source.

ZZZ contains multiple files including FLFIFS archives of different languages. The ZZZ header starts with the number of entries followed by FileData entries. File Source functions can accept FileData entries to extract from ZZZ files.

References:

Archives¶

The Archives object plays a crucial role in locating and organizing archives within a specified directory. Its primary function involves indexing these archives and facilitating a straightforward interface for reading the contained files.

struct Archives¶

Reads a path looking for ZZZ or FIFLFS archives. It remembers the locations of the archives, and serves serves as a front end to access them.

Public Functions

template<ArchiveTypeT archiveTypeT> inline requires valid_archive_type_t< archiveTypeT > const auto & get () const noexcept

Get archive via ArchiveTypeT

Template Parameters:: archiveTypeT –
Returns:

template<ArchiveTypeT archiveTypeT> inline requires valid_archive_type_t< archiveTypeT > auto & get () noexcept

Get archive via ArchiveTypeT

Template Parameters:: archiveTypeT –
Returns:

inline explicit operator bool() const noexcept¶

convert to bool

Returns:

inline auto get_nested(const std::initializer_list<std::string_view> &nested_archive) const¶

Gets the nested FIFLFS files from Field archive.

Parameters:: nestedArchive – string to filter results. {} will get all nested archives.
Returns:: all results.

Archives() = default¶: This will be in an invalid state but it’s so I can default construct and move in valid values later.

inline Archives(const std::filesystem::path &path, std::string_view lang)¶

Preloads all archives in the path.

Parameters:: path – that contains FIFLFS files or ZZZ files.

template<std::intmax_t minT = static_cast<std::intmax_t>(ArchiveTypeT::begin), std::intmax_t maxT = static_cast<std::intmax_t>(ArchiveTypeT::end), typename lambdaT> inline requires valid_archive_type_t< minT > &&valid_archive_type_t< maxT, true > &&valid_lambda< lambdaT > bool loop (const lambdaT &lambda) const

Loop through each of the archives.

Template Parameters:

minT – min archive
maxT – max archive
lambdaT – type of lambda function that returns bool and takes archive types. The bool returned short circuits the loop.

Parameters:

lambda – lambda of type lambdaT

Returns:

true if ran to completion

template<ArchiveTypeT... aT, typename lambdaT> inline requires valid_archive_type_t_v< aT... > bool specify (const lambdaT &lambda)

run lambda on listed types.

Template Parameters:

aT –
lambdaT –

Parameters:

lambda –

Returns:

inline bool test_set() const¶

if true all archives are valid.

Returns:

template<ArchiveTypeT... aT> inline requires not_zero< aT... > bool test_set ()

if true listed archives are valid

Template Parameters:: aT –
Returns:

template<bool nested = true, typename lambdaT, typename filterT = decltype(default_filter_lambda)> inline requires valid_execute_on_lambda< lambdaT > &&valid_filter_lambda< filterT > bool execute_on (const std::initializer_list< std::string_view > &filename, lambdaT &&lambda, filterT &&filter_lambda={}) const

execute on all archives the following lambda.

Template Parameters:

nested –
lambdaT –

Parameters:

filename –
lambda –

Returns:

template<bool nested = true, ArchiveTypeT... aT, typename lambdaT, typename filterT = decltype(default_filter_lambda)> inline requires not_zero< aT... > &&valid_execute_on_lambda< lambdaT > &&valid_filter_lambda< filterT > bool execute_on (const std::initializer_list< std::string_view > &filename, lambdaT &&lambda, filterT &&filter_lambda={}) const

execute on all listed archives the following lambda.

Template Parameters:

nested –
aT –
lambdaT –

Parameters:

filename –
lambda –

Returns:

ArchiveTypeT¶

The ArchiveTypeT enum encapsulates identifiers for different archive types in Final Fantasy VIII. There are six primary FIFLFS archives: battle, field, magic, main, menu, and world. Additionally, there are two main zzz archives for the FF8 remaster: zzz_main and zzz_other. These enums serve as convenient references for utilizing the “get<>” function in archives, allowing users to specify the desired archive type when interacting with the archives in the game code. The enum also provides utility constants such as ‘count,’ ‘first,’ ‘last,’ ‘begin,’ and ‘end’ for easy enumeration management.

enum class open_viii::archive::ArchiveTypeT : std::uint8_t¶

There are 6 main FIFLFS archives for ff8 and 2 2 main zzz archives for ff8 remaster. These enums are for get<> in archives to ask for the one you want.

Values:

enumerator battle¶

enumerator field¶

enumerator magic¶

enumerator main¶

enumerator menu¶

enumerator world¶

enumerator zzz_main¶

enumerator zzz_other¶

enumerator count¶

enumerator first¶

enumerator last¶

enumerator begin¶

enumerator end¶

FIFLFS¶

The FIFLFS archive format in Final Fantasy VIII for PC consists of three main components: File Index (FI), File Source (FS), and File List (FL). FI stores offset and size information, FS contains raw bytes, and FL holds virtual file paths. These elements collectively structure the archive, facilitating organized storage and retrieval of game assets.

template<bool HasNested> struct FIFLFS : public open_viii::archive::FIFLFSBase¶

A class representing a FIFLFS archive.

Forward declaration of the FIFLFS template struct.

This class provides methods to access and manipulate FIFLFS archives, with support for both nested and non-nested archives.

Template Parameters:: HasNested – A boolean flag indicating whether the archive has nested archives.

Grouping¶

Grouping is essentially an object that stores information about different components of the archive, serving as a reference for later reading. This includes details like file paths and offsets, enabling easy retrieval and interpretation.

template<std::ranges::contiguous_range T> struct Grouping¶

Grouping contains the location of the file and/or the raw data.

Public Functions

inline const std::filesystem::path &path() const noexcept¶

get path to file containing archive

Returns:

inline std::size_t offset() const noexcept¶

get offset in bytes to start

Returns:

inline std::size_t size() const noexcept¶

get Size of file / also defaults size if value is 0.

Returns:: size_t

inline const T &data() const noexcept¶

get loaded data buffer

Returns:

inline const std::string &base() const noexcept¶

stem of filename upper cased

Returns:

inline const std::filesystem::path &nested_path() const noexcept¶

get path inside file

Returns:

inline explicit operator bool() const¶

convert to bool

Returns:: true means value is set and loaded.

File List¶

The File List (FL) consists of entries with virtual paths starting with c:\ff8. Each path is separated by a new line in Windows format (CRLF), maintaining the same order as the File Index (FI) entries.

namespace fl¶

Functions

inline constexpr void clean_path_string(std::string &input) noexcept¶

FL files contain internal file structure paths. As a flat text file. This class is used to search the strings for a filename or grab all the entries. The entry will be a string paired with an int that is the line number. This is used to ID the FI entries.

Todo:: refactor this to reduce parameters and reduce functions.

Remove the C:\ from the start, remove the \r from the end, and change \ to the correct slash. added skipFixed if data is set then i probably fixed slashes already.

Parameters:

input – updates this string
skipFixed – if false skip removing the \r from end and skip replacing slashes.

Returns:

void

inline std::string clean_path_string(std::string &&input) noexcept¶

Remove the C:\ from the start, remove the \r from the end, and change \ to the correct slash. added skipFixed if data is set then i probably fixed slashes already.

Parameters:

input – updates this string
skipFixed – if false skip removing the \r from end and skip replacing slashes.

Returns:

modified input

inline std::string clean_buffer(std::string &&in_buffer)¶

Take out carriage returns and replace slashes.

Parameters:: in_buffer – multi line string of paths.
Returns:: cleaned.

inline constexpr std::size_t get_max(const std::size_t &count, const std::size_t &limit)¶

Decide how much to reserve based on known count or a set limit.

Parameters:

count – computed max count
limit – manual limit placed

Returns:

0 or count or limit;

inline void sort_entries(std::span<std::pair<std::uint32_t, std::string>> vector)¶

Sort the strings. to make it easier to choose the correct string first. shorter length and then what ever str < str2 does.

Parameters:: vector – pairs of ints and paths

inline auto sort_entries(std::vector<std::pair<std::uint32_t, std::string>> &&vector)¶

Sort the strings. to make it easier to choose the correct string first. shorter length and then what ever str < str2 does.

Parameters:: vector – pairs of ints and paths
Returns:: sorted vector

inline std::vector<std::string> get_all_entry_strings(const tl::read::input &cont, const size_t &offset, const size_t &size = 0U, const size_t &count = 0U, const std::initializer_list<std::string_view> &needle = {}, const size_t &limit = 0U)¶

Eagerly populate a vector with strings of paths, then sort it.

Todo:: make needle a predicate lambda.

Note

If size, limit, count are all 0 it’ll read till end of file. Or till it reads an empty line.

Parameters:

cont – wrapper on istream or span. For the source data.
offset – to start of data.
size – + offset is the end of the data.
count – max count detected from FI filesize / 12U.
needle – set of strings to search for.
limit – manually set max count.

template<typename path_t> inline auto get_all_entry_strings (const path_t &path, const size_t &offset, const size_t &size=0U, const size_t &count=0U, const std::initializer_list< std::string_view > &needle={}, const size_t &limit=0U) requires(std

template<typename path_t> inline std::vector< std::string > get_all_entry_strings (const path_t &path, const std::string &data, const std::size_t &offset, const std::size_t &size=0U, const std::size_t &count=0U, const std::initializer_list< std::string_view > &needle={}, const std::size_t &limit=0U) requires(std

Get All entries sorted from file or data buffer.

Parameters:

path – filename path.
data – buffer of bytes.
offset – bytes from start of data to start looking.
size – of FL file if known; 0 == unlimited
count – expected number of matches; calculated from FI file size / 12; 0 == unlimited
needle – possible string matches; {} == all
limit – max matches; 0 == unlimited

Returns:

matches

inline std::vector<std::pair<std::uint32_t, std::string>> get_all_entries(const tl::read::input &cont, const size_t &offset, const size_t &size = 0U, const size_t &count = 0U, const std::initializer_list<std::string_view> &needle = {}, const size_t &limit = 0U)¶

Eagerly populate a vector with pairs of (id,path), then sort it.

Todo:: make needle a predicate lambda.

Note

If size, limit, count are all 0 it’ll read till end of file. Or till it reads an empty line.

Parameters:

cont – wrapper on istream or span. For the source data.
offset – to start of data.
size – + offset is the end of the data.
count – max count detected from FI filesize / 12U.
needle – set of strings to search for.
limit – manually set max count.

template<typename path_t> inline auto get_all_entries (const path_t &path, const size_t &offset, const size_t &size=0U, const size_t &count=0U, const std::initializer_list< std::string_view > &needle={}, const size_t &limit=0U) requires(std

inline std::vector<std::pair<std::uint32_t, std::string>> get_all_entries(const std::string &data, const size_t &offset, const size_t &size, const size_t &count, const std::initializer_list<std::string_view> &needle, const size_t &limit)¶

inline std::vector<std::pair<std::uint32_t, std::string>> get_all_entries(const std::filesystem::path &path, const std::string &data, const size_t &offset, const size_t &size = 0U, const size_t &count = 0U, const std::initializer_list<std::string_view> &needle = {}, const size_t &limit = 0U)¶

Get All entries sorted from file or data buffer.

Parameters:

path – filename path.
data – buffer of bytes.
offset – bytes from start of data to start looking.
size – of FL file if known; 0 == unlimited
count – expected number of matches; calculated from FI file size / 12; 0 == unlimited
needle – possible string matches; {} == all
limit – max matches; 0 == unlimited

Returns:

matches

template<typename T> inline std::pair<std::uint32_t, std::string> get_entry(const T &data, const std::initializer_list<std::string_view> &needle, const size_t &offset = 0U, const size_t &size = 0U, const size_t &count = 0U)¶

Get a single entry that is the first match for needle.

Parameters:

data – contains buffer of chars
needle – is a group of strings to filter the output with.
offset – is the number of bytes to skip.
size – is max number of bytes. 0 is unlimited.
count – is max results returned. 0 is unlimited.

inline std::pair<std::uint32_t, std::string> get_entry(const std::filesystem::path &path, const std::string &data, const std::initializer_list<std::string_view> &needle, const size_t &offset = 0U, const size_t &size = 0U, const size_t &count = 0U)¶

Get a single entry that is the first match for needle.

Parameters:

path – contains path to file
data – contains buffer of chars
needle – is a group of strings to filter the output with.
offset – is the number of bytes to skip.
size – is max number of bytes. 0 is unlimited.
count – is max results returned. 0 is unlimited.

Variables

static constexpr auto EXT = std::string_view(".fl")¶: File extension

File Index¶

The File Index (FI) contains information such as the offset, uncompressed file size, and compression type, with each entry occupying 12 bytes. To determine the total number of files, divide the size of the FI file by 12.

struct FI¶

FI is the file index for the FL and FS files.

Public Functions

inline constexpr auto uncompressed_size() const noexcept -> std::uint32_t¶

Get the uncompressed size of the data represented by this block.

Returns:: The size of the block’s data after decompression, in bytes.

inline constexpr auto offset() const noexcept -> std::uint32_t¶

Get the offset of the block within the compressed file.

Returns:: The offset in bytes from the beginning of the compressed file where this block’s compressed data can be found.

inline constexpr auto compression_type() const noexcept -> CompressionTypeT¶

Get the compression algorithm used to compress this block’s data.

Returns:: The type of compression used to compress the data in this block, as defined by the CompressionTypeT enum. Possible values are none (for uncompressed data), lzss, or lz4.

Public Static Attributes

static constexpr auto SIZE = std::size_t{12U}¶

The size of the FI struct in bytes.

This constant stores the expected size of the FI struct, in bytes. It is used in a static assertion to ensure that the size of the FI struct matches the expected size at compile time.

static constexpr auto EXT = std::string_view{".fi"}¶

The file extension for files containing FI data.

This constant stores the file extension for files containing FI data. The extension is stored as a std::string_view for efficiency and ease of use.

File Source¶

File Source (FS) files store raw bytes of entries. For LZSS entries, the file begins with 4 bytes indicating the compressed file size. L4Z entries have a 12-byte header, including section size, an unused 4 bytes, and uncompressed size. The compressed size can be calculated by subtracting 8 from the section size.

namespace FS¶

Functions

template<is_default_constructible_has_data_size_resize outputT = std::vector<char>, typename input_t = tl::read::input> static outputT get_entry_lzss (input_t &input, const std::uint32_t uncompressed_size) requires(std

Get entry and uncompress via lzss

Note

not meant to be used directly

Note

lzss doesn’t need the uncompressed size to extract the data but it’s used to reserve the memory before uncompressing.

Template Parameters:

outputT – type to be returned: example std::string, std::vector<char>
fiT – FI or FI compatible type.

Parameters:

input – buffer adapter that holds a std::span<char> or a std::istream
uncompressed_size – number of bytes expected to expand to.

Returns:

output() filled with uncompressed data.

template<is_default_constructible_has_data_size_resize outputT = std::vector<char>, typename input_t = tl::read::input> static outputT get_entry_lz4 (input_t &input) requires(std

Get entry and uncompress via lz4

Note

not meant to be used directly

Note

L4Z header contains size of total section as uint32, 4 byte string, and a uint32 of the uncompressed size.

Template Parameters:

outputT – type to be returned: example std::string, std::vector<char>
fiT – FI or FI compatible type.

Parameters:

input – buffer adapter that holds a std::span<char> or a std::istream
uncompressed_size – number of bytes expected to expand to.

Returns:

output() filled with uncompressed data.

template<is_default_constructible_has_data_size_resize outputT = std::vector<char>, FI_Like fiT = FI, typename input_t = tl::read::input> static outputT get_entry (input_t input, const fiT fi, const std::size_t offset=0U) requires(std

Get entry and uncompress via lz4

Template Parameters:

outputT – type to be returned: example std::string, std::vector<char>
fiT – FI or FI compatible type.

Parameters:

input – buffer adapter that holds a std::span<char> or a std::istream
fi – contains compression type, uncompressed size, and offset
offset – additional offset value added to fi.offset()

Returns:

output() filled with uncompressed data.

template<is_default_constructible_has_data_size_resize outputT = std::vector<char>, FI_Like fiT = FI, typename path_t = std::filesystem::path> static outputT get_entry (const std::filesystem::path &path, const fiT &fi, const size_t &offset=0U) requires(std

get file entry and decompress it

Template Parameters:

outputT – type being returned
fiT – type of FI or FileData that contains offset, size, compression.

Parameters:

path – to file
fi – FI or FileData
offset – to file data in bytes

Returns:

uncompressed file

template<is_default_constructible_has_data_size_resize outputT = std::vector<char>, FI_Like fiT = FI, typename span_t = std::span<const char>> static outputT get_entry (span_t data, const fiT &fi, const size_t offset=0U) requires(std

get file entry and decompress it

Template Parameters:

outputT – type being returned
fiT – type of FI or FileData that contains offset, size, compression.

Parameters:

data – buffer with embedded file
fi – FI or FileData
offset – to file data in bytes

Returns:

uncompressed file

Variables

static constexpr auto EXT = std::string_view(".fs")¶

File Source

ZZZ¶

ZZZ files initiate with a 4-byte file count, representing the number of File Data entries that follow. All entries within a ZZZ file remain uncompressed. To access an FS file within the ZZZ file, combine a given FI offset with the corresponding File Data offset.

struct ZZZ¶

ZZZ file archive from FF8 remaster.

File Data¶

File Data entries feature variable sizes, requiring a one-at-a-time reading approach. To streamline this process, caching is employed. Each entry commences with a 32-bit size of the virtual file paths, where these paths are relative, such as data\disk\disk1. The entry then includes the corresponding virtual file paths, a 64-bit file offset, and a 32-bit file size.

struct FileData¶

FileData is an FI Like that contains the filepath, offset and size.

Public Functions

inline FileData(decltype(m_filename) filename, const std::unsigned_integral auto offset, const std::unsigned_integral auto size)¶

Piecemeal constructor

Parameters:

filename – path to file
offset – path to
size –

inline constexpr auto total_size() const noexcept¶

size of this file entry in the zzz file. This is for appending a new entry. I don’t know if it works.

Todo:: used in tests no where else. might want to make this a free function of the tests.

Returns:

inline auto get_path() const¶: gets path as a std::filesystem::path

inline constexpr auto uncompressed_size() const noexcept¶: alias for Size that should mirror FI

inline constexpr auto offset() const noexcept¶: get offset of file

inline auto get_path_string_view() const¶: gets path as a std::string_view

inline auto get_path_string() const¶: gets path as a std::string

template<std::size_t I> inline auto get() const noexcept¶: gets member variables

Note

required to structured binding support

Public Static Functions

static inline constexpr auto compression_type() noexcept¶

Compression type required to match FI like concept.

Returns:: no compression