Now, all of the 'to' site fields in filtering rules can specify a port,
not just the dstip sites.
Fix the precedence of sites in the same type of rules. For example, if
we find a match with an sni site, we should not stop searching for a
match in cn, because a matching cn site may have a higher precedence
than the matching sni site. We should apply the action of the cn site,
although sni rules have precedence over cn. The same applies to http
host and uri rules too.
Fix the precedence of dstip rules.
Improve and update unit and e2e tests accordingly.
Now, the filter uses B-trees for exact string matching and Aho-Corasick
machines for substring matching. B-trees and AC machines are exported to
linked lists for debug logging only.
Also,
- Separate all_sites and all_ports filters from substring filters. They
are not related with substring filters actually, and ACM keywords cannot
be empty strings anyway. So now they should be handled separately too.
- Improve debug logging of filtering rules.
- Update unit tests accordingly, and improve.
- Fix pxyconn_filter(), keep searching for a match in substring filters
if exact match does not have a matching site rule.
- Increase common names max len and tokens. weather.gov has 73 tokens.
- Rename keyword to desc.
- Update documentation.
- Clean up.
Actually, no need to check all_sites or all_ports, because strstr(3) on
OpenBSD reads that "If little is an empty string, big is returned", and
if all_sites or all_ports is set, site or port (little/needle) is empty.
But using all_sites and all_ports should improve performance by avoiding
the strstr() call.
Add to the end of linked lists for correct list ordering, but btrees
cannot obey this ordering.
Also, update the unit tests accordingly.
And fix compile with WITHOUT_USERAUTH.
So, for 'to' fields too, we use two separate data structures: binary
search trees (BST) for exact match and linked lists for substring match.
Now all 'from' and 'to' fields in filtering rules use these two data
structures.
To repeat, filtering rules should be written with exact matches instead
of substring matches, as much as possible. Because BST search must be
much faster than substring search over linked lists.
To repeat, we have modifed kbtree to support complex data structures in
from fields.
Also, update the unit tests accordingly.
So, now we use two separate data structures: binary search trees (BST)
for exact match and linked lists for substring match.
Currently, only user, keyword, and ip 'from' fields in filtering rules
use these two data structures. This also means that now we support exact
and substring matches in 'from' fields.
Filtering rules should be written with exact matches instead of
substring matches, as much as possible. Because BST search must be much
faster than substring search over linked lists.
We have modifed kbtree to support complex data structures in from
fields.
all_sites and all_ports rules should be at the end of their lists, they
should be searched last, because they are the least specific rules in
their lists, hence have lower precedences.
Also, obey the order of rules in conf files by adding sites, ports, and
macro values to their lists in the same order they are in conf files.
Update the unit and e2e tests accordingly, and improve.
Now the target IP address filters can use port specs too.
Refactor for code reuse, create filter_action struct used by rules,
sites, and ports.
Also, improve code and documentation.
End-to-end tests now require testproxy v0.0.4, which supports the new
Reconnect command for the Pass filtering rule.
Split mode with the -n option also supports filtering rules, so the
Divert rule can enable the divert mode even with the -n option. This is
because the purpose of the -n option is to convert sslproxy into an
sslsplit, and we want to support filtering rules in sslsplit-like
sslproxy too.
Do not call protopassthrough_engage() twice. If we call it to apply the
deferred pass action and also set the ctx->pass flag, it will be called
again in protossl_setup_src_ssl(). So, just set the flag and leave the
rest to protossl_setup_src_ssl().
Return value of 1 is for ending macro expansion. Otherwise, it coincided
with closing brace retval, which was wrongly breaking out of the while
loop in load_proxyspec_struct().
TODO: Use enums for return values.
It is better to error out earlier than to pass the option down to be
rejected by its handler (the handler may fail to reject, as was the case
with a couple of options).
Fix possible segfault if name has leading white space
Pass the name param to get_name_value() as char *, so it cannot be
modified ever
Improve unit tests for get_name_value and proxyspec_parse
And use all those return values.
Since we support include files now, we should be able to report in which
include file the error has occured. This is not possible if functions
just bail out calling exit(), because the user has to scroll back stderr
lines to find which include file has failed loading (a line starting
with 'Conf: ').
Plus, calling exit() on errors reduces unit testability of functions.
Also, handle all possible out of memory conditions in opts.c.
The new Define option can be used for defining macros to be used in
filtering rules. Macro names must begin with a '$' char. Macro values
must be separated with spaces.
Macros are expanded by rewriting the rule with the values of macro.
PassSite rules do not support macros (the PassSite option will be
deprecated in favor of filtering rules in the future).
Filtering rules can enable/disable or don't change logging. If a rule
does not mention a log action, its logging should not change. So, binary
log action fields were not enough to represent those 3 possibilities,
hence we have increased the size of those fields to 2-bits.
We should obey the order of rules as they are written in the conf file,
because latter rules should be able to override the log actions of
earlier rules. So, keep the order.
We should defer pass and/or block actions as long as possible, because a
higher precedence rule in SSL filter should be able to override (cancel)
deferred pass and block actions taken by a lower precedence rule in Dst
Host filter. And in HTTP filter the same applies to deferred block
actions taken by Dst Host and SSL filters.
Also, thanks to this new deferred actions, now HTTP filter can keep
enabled divert and split modes. In other words, a higher precedence HTTP
filter rule can cancel a deferred block action set by a lower precedence
rule earlier, which was not possible before without deferred actions and
rule precedence.
And other improvements.
Now filtering rules can disable log actions too. This is possible thanks
to the newly added precedence field of rules. Log actions of filtering
rules at higher precedence can modify logging now. In other words, more
specific rules can change the log actions of more general rules.
HTTP filtering rules can only disable logging.
Now we assign precedence to each filtering rule. More specific rules
have higher precedence. So, filtering rules at lower precedence cannot
override the actions applied to a conn by filtering rules at higher
precedence.
The other precedence rules still apply.
Log actions specified in HTTP filter rules can never enable disabled
logging, because their loggers would not be initialized.
Perhaps we should initialize them in the log submit function, if they
are initialized yet.
