gul17/tokenize.h

Detailed Description

Split a string into substrings.

Functions
template<typename StringContainer = std::vector<std::string>, typename ContainerInsertFct = void (*)(StringContainer&, std::string_view)>
StringContainer	gul17::tokenize (std::string_view str, std::string_view delimiters=default_whitespace_characters, ContainerInsertFct insert_fct=detail::emplace_back< StringContainer >)
	Split the given string into a vector of substrings (tokens) delimited by any of the characters in the delimiters string.
template<typename StringContainer = std::vector<std::string_view>, typename ContainerInsertFct = void (*)(StringContainer&, std::string_view)>
StringContainer	gul17::tokenize_sv (std::string_view str, std::string_view delimiters=default_whitespace_characters, ContainerInsertFct insert_fct=detail::emplace_back< StringContainer >)
	Split the given string into a vector of substrings (tokens) delimited by any of the characters in the delimiters string.

Function Documentation

tokenize()

template<typename StringContainer = std::vector<std::string>, typename ContainerInsertFct = void (*)(StringContainer&, std::string_view)>

StringContainer gul17::tokenize ( std::string_view  str, std::string_view  delimiters = default_whitespace_characters, ContainerInsertFct  insert_fct = detail::emplace_back<StringContainer>  )

inline

Split the given string into a vector of substrings (tokens) delimited by any of the characters in the delimiters string.

Multiple adjacent delimiters are treated like a single one, and delimiters at the beginning and end of the string are ignored. For example, tokenize(" A B C ") yields a vector with the three entries "A", "B", and "C".

// Default return type std::vector<std::string>
auto tokens = tokenize("  Hello world  ");
assert(tokens.size() == 2);
assert(tokens[0] == "Hello");
assert(tokens[1] == "world");

This function returns a std::vector<std::string> by default, but a compatible container for string/string_view types can be specified via a template parameter:

Template Parameters

StringContainer	A container for strings or std::string_view-like types, e.g. std::vector<std::string> or std::list<std::string_view>
ContainerInsertFct	Type for the insert_fct function parameter.

Parameters

str	The string to be split.
delimiters	String with delimiter characters. Any of the characters in this string marks the beginning/end of a token. By default, a wide variety of whitespace and control characters is used.
insert_fct	By default, tokenize() calls the emplace_back() member function on the container to insert strings. This parameter may contain a different function pointer or object with the signature void f(StringContainer&, std::string_view) that is called instead. This can be useful for containers that do not provide emplace_back() or for other customizations.

Returns

a container with the single substrings.

// Return std::string_views instead of strings (like tokenize_sv())
auto parts1 = tokenize<std::vector<std::string_view>>("Hello world");
assert(parts1.size() == 2);
assert(parts1[0] == "Hello");
assert(parts1[1] == "world");
 
// Use a different container that provides emplace_back()
auto parts2 = tokenize<gul17::SmallVector<std::string_view, 3>>("a-b-c", "-");
assert(parts2.size() == 3);
assert(parts2[0] == "a");
assert(parts2[1] == "b");
assert(parts2[2] == "c");
 
// Use a different container with a custom inserter function
using WeirdContainer = std::queue<std::string>;
auto inserter = [](WeirdContainer& c, std::string_view sv) { c.emplace(sv); };
auto parts3 = tokenize<WeirdContainer>("a.b", ".", inserter);
assert(parts3.size() == 2);
assert(parts3.front() == "a");
assert(parts3.back() == "b");

Note

tokenize() does not assume a specific encoding for its input strings, but operates on individual chars. This can have surprising effects in code such as this:

auto words = tokenize("Hörgeräteakkustiker hätten es gewußt", "ä");
assert(words.size() == 3); // Might fail or succeed depending on the encoding

See also

gul17::tokenize_sv() returns a vector<std::string_view> by default, gul17::split() uses a different approach to string splitting.

Since

GUL version 2.5, the return type of split() can be specified as a template parameter and a custom inserter can be specified (it always returned std::vector<std::string> before).

References gul17::bit_set().

tokenize_sv()

template<typename StringContainer = std::vector<std::string_view>, typename ContainerInsertFct = void (*)(StringContainer&, std::string_view)>

StringContainer gul17::tokenize_sv ( std::string_view  str, std::string_view  delimiters = default_whitespace_characters, ContainerInsertFct  insert_fct = detail::emplace_back<StringContainer>  )

inline

Split the given string into a vector of substrings (tokens) delimited by any of the characters in the delimiters string.

This function is identical to tokenize(std::string_view, std::string_view, ContainerInsertFct) except that it returns a std::vector of std::string_views instead of strings by default:

auto tokens = tokenize_sv("hello world", " "); // Return type is std::vector<std::string_view>
assert(tokens.size() == 2);
assert(tokens[0] == "hello");
assert(tokens[1] == "world");

See also

gul17::tokenize() returns a vector<string> by default, gul17::split() uses a different approach to string splitting.

Since

GUL version 2.5, the return type of tokenize_sv() can be specified as a template parameter and a custom inserter can be specified (it always returned std::vector<std::string_view> before).

References gul17::bit_set().